WSO2 Enterprise Service Bus (ESB) / Apache Synapse: Time to be lightweight and fast!
We compared the performance of the WSO2 ESB / Apache Synapse framework with two open source alternatives this time, namely Apache ServiceMix 3.1 and Mule 1.4.1. We tested three simple and common scenarios, namely:
- The ability to create proxy services with a custom interface that proxies requests to a backend service
- The ability to perform content based routing on a given XPath expression over the payload
- The ability transform the incoming and outgoing messages through XSLT
The WSO2 ESB clearly leads in all comparisons, followed closely by the proprietary ESB that we compared against last time. Third is Apache ServiceMix followed by Mule. However, we encountered many problems getting even the above scenarios configured through Mule, and writing to the Mule developer or user lists didn't yield good answers probably because those are known issues with Mule.
The first problem we encountered was that Mule was unable to create proxy services at all. Then we saw that it was having quite a few problems when keepalive connections were used - both by clients, and from it to its backend services. It was also sending an unwanted HEAD request before a POST which was quite irritating, and it couldn't be turned off either. When we were trying out transformations, Mule kept failing on every other request.
We were only able to compare the performance of Mule vs the WSO2 ESB / Synapse for the content based routing scenario, and the WSO2 ESB was over 4 times faster than Mule! Thats probably the reason why I haven't yet seen any detailed performance tests about Mule. Commenting on BEA's Dain Hansen's blog, Ross Mason of Mule said that they were working on publishing some figures by the end of the year 2006, but so far I have not been able to read about any.
Recently on TSS, Eugene Ciurana was claiming that Mule was able to do 200 million transactions a day, and that no other ESB they evaluated, commercial or open source, could do that on the same hardware. This means that no ESB was able to perform 2315 TPS? I wish they would describe the scenario tested and the hardware used, instead of claiming such numbers out loud. For example a single instance of the WSO2 ESB / Synapse could do well over 3200 TPS with 40 concurrent users for some scenarios we've tested, on standard hardware (see 'Notes' here). I am almost sure that Eugene is yet to see the WSO2 ESB /Apache Synapse in action, cause they are not like the good ol' Mules that supposedly "Just works"... they work much faster and smarter! and they are extremely lightweight.
References
[1] WSO2 ESB Performance Testing Round 1 [http://wso2.org/library/1721]
[2] WSO2 ESB Performance Testing Round 2 [http://wso2.org/library/2259]
- asankha's blog
- Login or register to post comments
- Printer friendly version
- 2092 reads
200M transactions - testing methodology
Howdy.
That was done on a dedicated X4200 from Sun, running Solaris and Java 5, with 8 GB RAM, multiple Mule instances. Gigabit connection over fibre between the Mule server and other endpoints. Each message flowing through the ESB was ~10KB, with inbount, service, and outbound router processing and transformations. The messages flowed through the ESB through two separate transports. The message sample was generated at random so that 33% of each matched:
* JMS - JMS endpoints (payload transformation only)
* SOAP - SOAP endpoints (payload transformation only)
* JMS - SOAP endpoints (payload and envelope transformations,
multiple output routes)
SEDA for the Mule container; TIBCO JMS and SOAP for Xfire. Logging was enabled in all cases for analysis.
I discussed the stats and methodology at TheServerSide Java Symposium presentation, though I may have omitted to present them for the discussion on TSS afterward. Here they are, though, and feel free to ask if you have further questions.
Due to NDAs, I can't disclose the commercial ESBs that we tested; however, the open-source ESBs included ServiceMix and OpenESB. For commercial, assume all the usual suspects, minus the guys who invented the ESB concept (they wanted to create vendor lock-in *before* we even tested). The commercial offering that came well in performance, service, interoperability, etc. was from a vendor that sounds like BITCO; at the end the decision to go with Mule was economical: equivalent performance, lower price, equivalent service and support, but a few hundred thousand dollars cheaper. Either ESB would've done the job well. None of the others came close to the two finalists in terms of performance or robustness.
I began looking at your software on Sunday 15.July, on a recommendation from Joe Ottinger... so your comment was accurate as of 12.July!
Cheers,
Eugene
Non-stop action. A vulnerable hero. A quest to save the world. It's the
most exciting novel of the decade:
The Tesla Testament by Eugene Ciurana
ISBN: 1-4116-7317-4 - BISAC: FIC031000