Previously, we tested Apache Axis1/Java vs Axis2/Java using a modified version of the Indiana University SOAP benchmark. The tests were modified to move from rpc/encoded to be document/literal, because rpc/encoded has been deprecated by the WS-I Basic Profile. You can read about the previous benchmark tests here: http://wso2.org/library/91. For this test, we made one further improvement to the WSDL to sort out this problem.
In case you didn't read the previous test results, the test consists of a number of invocations against the server with an array of different types. Each invocation echoes the results back. The code written in each case uses the databinding framework to copy the data from the incoming objects to the outgoing objects, fully working the databinding framework. The types include: void, ints, doubles, strings, floats. There are also two complex types that simulate more interesting workloads -
MeshInterface. For each type, we tested various sizes of array by increasing the numbers of elements from 1 to 5000. In the previous tests we analyzed read/write and echo performance. This time we focussed on echo in order to keep the permutations down.
Since Apache Axis2/Java has been updated, it seemed like a good time to do some more performance benchmarking. To make it more interesting we changed the comparison point to be the Codehaus XFire Web Services toolkit. Codehaus XFire is also based on the StAX parser model. In fact both XFire and Axis2 use the same parser (Woodstox) by default. The XFire team pride themselves on the performance of the toolkit (for example see http://xfire.codehaus.org/Performance).
Both Codehaus XFire and Apache Axis2/Java support a number of databinding frameworks. Because this benchmark is driven from WSDL, we chose binding frameworks that support simple WSDL first tooling. For this scenario, The XFire toolkit recommmends using JAXB or XMLBeans when doing WSDL-driven development. We chose to use the JAXB 2.0 RI, given that our previous experience with XMLBeans was that it wasn't the most performant option.
JAXB support in Axis2 is still very new, and the default approach for WSDL-driven databinding in Axis2 is ADB (Axis2 Data Binding). Therefore, we tested both ADB and JAXB for Axis2. Because of the experimental nature of the JAXB support in Axis2 we didn't expect great things from the JAXB binding.
The performance tests we ran, only test the server-side capabilities of the products. We ran both products under the same instance of Apache Tomcat (5.5.20) running on Sun's Java JDK 1.5.0_10-b03. The Java options used were:
JAVA_OPTS='-server -Xms2g -Xmx2g -Xss512k -XX:PermSize=512m -XX:MaxPermSize=512m
-XX:+UseParallelGC -XX:+UseParallelOldGC -XX:+AggressiveOpts -XX:+UseBiasedLocking'
The server was running on Fedora Core 5 on a 4-way Xeon 3.2Ghz server with 2GB RAM. The client machine was connected to the server via a dedicated 1Gb Ethernet switch. As a client driver we used the Apache Bench tool shipped with the Apache HTTPD server. This tool makes HTTP POST commands using multiple threads. For the tests we used 25 client threads to simulate a high-load environment. In order to have a reasonable run through, we ran each test with 2500 requests giving each thread 100 invocations. In order to warm up the JIT and achieve a reasonable running state, we ran through the tests on both frameworks once and threw the results away before capturing the results.
The client driver was running on an identical box with Microsoft Windows Server 2003 R2 Enterprise Edition (Service Pack 1).
See Resources for the benchmark code .
Axis2 using the default ADB binding framework shows outstanding performance, with consistently better results than XFire/JAXB or Axis2/JAXB. XFire/JAXB is consistently better than Axis2/JAXB. Axis2/ADB scales the best as the dataset size increases, and Axis2/JAXB scales the worst as the dataset size increases. Based on this data we plan to look at Axis2/JAXB performance.
You may look at these results and say "XFire outperforms Axis2 when they use the same databinding toolkit", and that would be true, but not necessarily a useful comparison. The JAXB support in Axis2 is still classed as experimental. Given that many Web services developers use the WSDL-first model, then a fair comparison would be to look at the recommended data-binding in each case. Given that premise, Axis2 is on average at least 40% faster than XFire over all the tests we did.
Another key point to make is that these results clearly show that Web Services are not slow. Using either toolkit, the overhead of using XML and SOAP is no longer a limiting factor in writing distributed systems for most applications (with may be the exception of trading floors!). While these tests do not perform 'real' work, the fact that a XML messaging system can scale to more than 10 million transactions an hour on a single quad-core server shows that Web services can be used for significant systems applications.
The first chart shows the results for Ints, Doubles, Strings, SimpleEvents and MeshInterface, all at an array size of just 1 element. The Y-Axis shows the number of requests per second throughput. At this size, there is very little between the engines or databinding, showing that they all cope well with the SOAP envelope processing. As well as the comparative performance, it is worth noting that all of these tests are offering remarkable performance. These engines are managing to handle more than 3,000 reqs/sec on a quad-core system. Over the course of a day, that is more than 250 million transactions.
The next chart shows the performance at array size 100. As you can see, this starts to stress the engines more effectively - bringing out the core differences in processing. In this scenario the message sizes range from 4-6k per invocation, and so this simulates doing some serious work - especially if we look at the complexType scenarios.
In these tests, Axis2/ADB has a significant advantage - on average 50% faster than XFire/JAXB, which is in turn 50% faster than Axis2/JAXB.
Note that at these levels, there is serious network utilization going on. There were times during the test when Axis2 was saturating the dedicated 1Gb/s Ethernet segment that the test was run on. The next chart shows network throughput (as measured by ApacheBench) for echoStrings.
The final graph shows the results for the echoMeshInterface test at different array sizes from 1 to 5000. This clearly shows how the engines stand up to increased load with complex object types. The first thing to notice is how well Axis2/ADB scales from 1 through 10 to 50 elements. The other thing to note is that for the larger size payloads (50 up) Axis2/ADB consistently performs 70% faster than XFire/JAXB.
It is clear that we have plenty of work to do still - in particular to optimize Apache Axis2 for JAXB. We also intend to test the POJO-first binding of Axis2 against the XFire's Aegis binding. That said, we can say that these results are simply excellent. They demonstrate that modern Web Services engines can perform at very high transaction rates: high enough to saturate a 1Gb/s Ethernet segment. In the specific case of contract-first development (where you start with WSDL) Apache Axis2/Java is significantly faster than Codehaus XFire if you choose the recommended bindings.