We compared the performance of the WSO2 ESB / Apache Synapse framework with two open source alternatives this time [1], namely Apache ServiceMix 3.1 and Mule 1.4.1. We tested three simple and common scenarios, namely:
- The ability to create proxy services with a custom interface that proxies requests to a backend service
- The ability to perform content based routing on a given XPath expression over the payload
- The ability transform the incoming and outgoing messages through XSLT
The WSO2 ESB clearly leads in all comparisons, followed closely by the proprietary ESB that we compared against last time [2]. Third is Apache ServiceMix followed by Mule. However, we encountered many problems getting even the above scenarios configured through Mule, and writing to the Mule developer or user lists didn't yield good answers probably because those are known issues with Mule.
The first problem we encountered was that Mule was unable to create proxy services [3] at all. Then we saw that it was having quite a few problems when keepalive [4] connections were used - both by clients, and from it to its backend services. It was also sending an unwanted HEAD [5] request before a POST which was quite irritating, and it couldn't be turned off either. When we were trying out transformations, Mule kept failing on every other request [6].
We were only able to compare the performance of Mule vs the WSO2 ESB / Synapse for the content based routing scenario, and the WSO2 ESB was over 4 times faster than Mule! Thats probably the reason why I haven't yet seen any detailed performance tests about Mule. Commenting on BEA's Dain Hansen's blog [7], Ross Mason of Mule said that they were working on publishing some figures by the end of the year 2006, but so far I have not been able to read about any.
Recently on TSS, Eugene Ciurana was claiming that Mule was able to do 200 million [8] transactions a day, and that no other ESB they evaluated, commercial or open source, could do that on the same hardware. This means that no ESB was able to perform 2315 TPS? I wish they would describe the scenario tested and the hardware used, instead of claiming such numbers out loud. For example a single instance of the WSO2 ESB / Synapse could do well over 3200 TPS with 40 concurrent users for some scenarios [9] we've tested, on standard hardware (see 'Notes' here [10]). I am almost sure that Eugene is yet to see the WSO2 ESB /Apache Synapse in action, cause they are not like the good ol' Mules that supposedly "Just works"... they work much faster and smarter! and they are extremely lightweight.
References
[1] WSO2 ESB Performance Testing Round 1 [http://wso2.org/library/1721 [11]]
[2] WSO2 ESB Performance Testing Round 2 [http://wso2.org/library/2259 [12]]