ESB stops accepting requests

I would first suspect deficiencies of ab for SOAP load testing. It cannot properly support HTTP 1.1 (chunking etc) and even has issues when https is used. Follow the OS tuning as well and use the Java Bench which is an equivalent to ab as mentioned in the ESB Performance benchmark series (of three parts) starting from http://wso2.org/library/1721 asankha http://esbmagic.blogspot.com

Well, I think ab is not the problem. If I use ab against the service it works like a charm:   $ ab -k -T 'text/xml; charset=UTF-8' -H 'SOAPAction: "http://tempuri.org/HelloWorld"' -p data.xml -n 1000 -c 10 http://testserver3/Services/Service1.asmxThis is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/Copyright 2006 The Apache Software Foundation, http://www.apache.org/Benchmarking testserver3 (be patient)Completed 100 requestsCompleted 200 requestsCompleted 300 requestsCompleted 400 requestsCompleted 500 requestsCompleted 600 requestsCompleted 700 requestsCompleted 800 requestsCompleted 900 requestsFinished 1000 requestsServer Software:        Microsoft-IIS/5.1Server Hostname:        testserver3Server Port:            80Document Path:          /Services/Service1.asmxDocument Length:        363 bytesConcurrency Level:      10Time taken for tests:   0.443336 secondsComplete requests:      1000Failed requests:        0Write errors:           0Keep-Alive requests:    1000Total transferred:      617000 bytesTotal POSTed:           533280HTML transferred:       363000 bytesRequests per second:    2255.63 [#/sec] (mean)Time per request:       4.433 [ms] (mean)Time per request:       0.443 [ms] (mean, across all concurrent requests)Transfer rate:          1357.89 [Kbytes/sec] received                        1174.69 kb/s sent                        2533.79 kb/s totalConnection Times (ms)              min  mean[+/-sd] median   maxConnect:        0    0   0.0      0       0Processing:     2    3   1.7      4      15Waiting:        2    3   1.7      3      15Total:          2    3   1.7      4      15Percentage of the requests served within a certain time (ms)  50%      4  66%      4  75%      5  80%      5  90%      6  95%      7  98%      8  99%      8 100%     15 (longest request)   I also tested the same thing with ALSB from BEA (now Oracle) with a proxy service for the same backend service, and it works as well. With WSO2 I had this strange issue that after some requests ab times out, so I created a file called <esbhome>\tomcat\lib\nhttp.properties with http.socket.timeout = 480 in it, and this somehow "fixed" the issue (can someone explain why?), but I get some real bad numbers: $ ab -k -T 'text/xml; charset=UTF-8' -H 'SOAPAction: "http://tempuri.org/HelloWorld"' -p data.xml -n 1000 -c 10 http://testserver2:8280/soap/HelloWorldThis is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/Copyright 2006 The Apache Software Foundation, http://www.apache.org/Benchmarking testserver2 (be patient)Completed 100 requestsCompleted 200 requestsCompleted 300 requestsCompleted 400 requestsCompleted 500 requestsCompleted 600 requestsCompleted 700 requestsCompleted 800 requestsCompleted 900 requestsFinished 1000 requestsServer Software: Synapse-HttpComponents-NIOServer Hostname: testserver2Server Port: 8280Document Path: /soap/HelloWorldDocument Length: 363 bytesConcurrency Level: 10Time taken for tests: 59.946698 secondsComplete requests: 1000Failed requests: 486 (Connect: 0, Length: 486, Exceptions: 0)Write errors: 0Non-2xx responses: 3Keep-Alive requests: 0Total transferred: 310410 bytesTotal POSTed: 526575HTML transferred: 188121 bytesRequests per second: 16.68 [#/sec] (mean)Time per request: 599.467 [ms] (mean)Time per request: 59.947 [ms] (mean, across all concurrent requests)Transfer rate: 5.05 [Kbytes/sec] received 8.58 kb/s sent 13.63 kb/s totalConnection Times (ms) min mean[+/-sd] median maxConnect: 0 0 0.3 0 3Processing: 12 594 399.1 229 2313Waiting: 0 111 108.8 207 300Total: 12 594 399.1 229 2313Percentage of the requests served within a certain time (ms) 50% 229 66% 999 75% 999 80% 1000 90% 1001 95% 1003 98% 1081 99% 1301 100% 2313 (longest request) the payload is the same, the server the ESBs run on in the same, the backend service is the same, but on Synapse I get 486 failed requests!!! and the longes request took 2313 ms for a simple Hello World proxy?! There has to be something fishy going on, and I would be happy to fix this issue. I would rather not change my benchmarking tool. ab does a simple POST request with the specified headers, like most SOAP clients.   Update: After some googling I think this issue was already raised: http://www.nabble.com/Intermittent-IO-Errors-using-Synapse-td9591696.html I'm going to test the ESB on a linux box and see if the problem persists.

First of all a Unix environment is highly recommended with a newer kernel, for optimal support for Java NIO From your run with ab, note the following: Keep-Alive requests: 0 AB was not able to use HTTP 1.1 keep alive with the server. In addition, HTTP 1.1 with chunking is much better than HTTP 1.0 with Content-Length encoding as the messages can then be streamed without buffering first to compute message length Finally, there have been some improvements done on the NIO transport lately, which is available with the Synapse trunk/nightly and this is also available for WSO2 support customers for the last release of v1.7.x asankha http://esbmagic.blogspot.com

First of all a Unix environment is highly recommended with a newer kernel, for optimal support for Java NIO Ok. From your run with ab, note the following: Keep-Alive requests: 0 AB was not able to use HTTP 1.1 keep alive with the server. Yes, I see. But that's not really a problem of ab, as it works fine with ALSB and IIS, so i guess that is a problem of NIO? In addition, HTTP 1.1 with chunking is much better than HTTP 1.0 with Content-Length encoding as the messages can then be streamed without buffering first to compute message length I'll try that too. Finally, there have been some improvements done on the NIO transport lately, which is available with the Synapse trunk/nightly and this is also available for WSO2 support customers for the last release of v1.7.x Well, we will be using this in production, so a nightly can not be used for now. BTW, thanks for this fantastic product (even if it is not always perfect :)). I will post my results once I get the problems sorted out.