Note: Please refer to the first article in this series at http://wso2.org/library/1721 for more information on the performance environment used, the sample services and the configuations.
In this round of tests, we compared the WSO2 ESB against two open source alternatives - Apache ServiceMix 3.1 and Mule 1.4.1. The WSO2 ESB is an Open Source project (under the Apache license) based upon the Apache Synapse project. You will recall from the first article that we had three specific benchmarks, designed to test the processing performance in some simple scenarios. These obviously aren't exhaustive and we understand there may be other scenarios where alternative benchmarks might show different results. However, these are common use-cases based on customer situations that we have encountered. The three scenarios are:
We faced a number of issues in implementing these benchmark scenarios inside Mule. We sent a number of requests for help to the Mule mailing lists, but in several cases we didn't receive any useful help. You can see the issues listed below with links to the mailing lists. Of course, we will be happy to re-run these tests again if at a later stage the issues are resolved.
We were able to implement all three scenarios with Apache ServiceMix without too much trouble. However, it is worth noting that we did encounter one significant runtime issue with ServiceMix. The WSO2 ESB uses a non-blocking transport based on Java NIO and the Apache HttpCore project. This means that the code can scale to handle thousands of concurrent requests while only using a limited threadpool. The ServiceMix transport seems to be based on blocking code which means that you can only handle as many concurrent requests as there are threads configured in the system. Our testing raised a significant problem that probably means that ServiceMix in its current state is not suitable for production use: when the number of concurrent requests exceeded the number of configured threads, ServiceMix crashes and needs to be restarted before it will handle further requests. For these tests we ensured that the configured number of threads was greater than the number of clients driving it. We have raised a JIRA against this with ServiceMix.
We created a service proxy on the ESB, to the SimpleStockQuote service, and used the client to request for quotes over the proxy service.
One aspect We wanted to test was getting the service proxy to show a different interface/WSDL for the service than the target/actual interface. This is important in the case where you wish to expose the same backend services over multiple proxies and/or interfaces. An example is versioning, where the proxy performs a transformation to the old or new format as necessary.
We did have problems with Apache ServiceMix configuring the proxy to present a WSDL of our choice to the client, even though we followed the instructions that claimed this support. (See wsdlResource in this page)
With Mule the problem was more serious: we were unable to create any kind of proxy or virtualization with the publicly available release. We wrote   to the Mule developer list but it seemed like there was only code attached to a JIRA issue, thats not yet available with the Mule distributions. Obviously we would have been unlikely to suggest this as a scenario if it didn't work with the WSO2 ESB! But, it's worth saying we did not come across any issues setting up a proxy service or publishing different WSDLs with the WSO2 ESB.
You can see the performance results here (obviously without any results for Mule). As you can see, the WSO2 ESB outperforms ServiceMix by 50-60% on this scenario. Although this is a very simple mediation, the results of over 2000tps are very impressive.
Unfortunately we once again had significant problems implementing this scenario in Mule.
We did not encounter any issues with Apache ServiceMix or the WSO2 ESB in setting up this scenario.
The graph below shows the results of the CBR scenario. As you can see, the WSO2 ESB was over 3 times faster than ServiceMix in this scenario. Once again, we managed incredibly high throughput (nearly 3000tps). The lead over Mule was even higher (over 400%) probably due to the problems Mule had with Keepalive and HEAD.
Once again, we encountered a serious issue with Mule: every other request failed - due to the inability of Mule to properly handle keepalive messages to the backend service. It seemed like Mule was using keepalive connections when forwarding the request to the backend service, but closed the connection before the second response could be written back.
Our initial ServiceMix tests used the default mode in XSLT called String mode. However, we found another mode called DOM as well, and we are not sure if there is a difference in these that would impact further processing of the message through ServiceMix. However, when using the String mode, ServiceMix beat the WSO2 ESB 1.0. This prompted us to review our XSLT performance, and we managed to do some further optimization. Thus we have included the results for the WSO2 ESB 1.0 and nightly build.
This set of results shows a comparison of the WSO2 ESB against two open source alternatives Apache ServiceMix and Mule, together with the data on the proprietary ESB we tested in the previous round.
For all three scenarios, the WSO2 ESB shows outstanding performance. ServiceMix does lead over the WSO2 ESB 1.0 in XSLT, however, the nightly build shows that with the correct optimizations, WSO2 retains the lead in all three scenarios.
Apart from the raw performance, we were concerned to find so many issues with Mule. The scenarios we attempted are really very simple and basic XML and HTTP use-cases. We were surprised at the difficulties we encountered and the problems we had resolving these issues on the Mule mailing lists.
In the scenario we did manage to get working, Mule was over 4 times slower than the WSO2 ESB and 3 times slower than the proprietary ESB
We were also concerned by the inability of ServiceMix to handle high concurrent workloads, and hope they can resolve this soon.
Overall, the results show that the core architecture of the WSO2 ESB and Apache Synapse is highly efficient. In particular, the non-blocking transport (based on Apache HttpCore) and streaming XML design (based on Apache Axiom project) mean that the system can handle very large numbers of concurrent connections and very high levels of transactions per second with very low memory and thread requirements.
It is always difficult to publish fair benchmarks - especially when your own initiative comprehensively beats all the other projects tested! This is made harder by the lack on any industry standard benchmark for ESBs (let alone a standard definition of an ESB). It is unlikely that this benchmark will be the last word in this space, and we look forward to collaborating with the other projects to build a set of open benchmarks in this space.
Asankha Perera, Architect, WSO2 Inc., asankha at wso2 dot com