WSO2 ESB QA Test Plan - WSO2 ESB

Introduction

This document defines the strategy and plan for testing a WSO2 ESB release. All QA testing activities during the release process will be based on this document.

Scope and Test Coverage

The WSO2 ESB is a lightweight, and high performance Enterprise Service Bus (ESB). Based on Apache Synapse and Apache Axis2 projects, it supports connectivity, transformation and mediation, and management of Web services interactions.

The following test types will be carried out during a QA testing cycle of the ESB

Installation/Smoke testing
UI/Usability testing
Functional testing
Regression testing

Feature list of the product for post 1.5 releases

The features which will be available since the 1.5 release will be as follows.

Proxy Services
Tasks
Endpoints
Sequences
Local Registry
Configuration
Integrated Registry (Ability to save/edit sequences and endpoints directly to/from the integrated registry)
System Information
Statistics
System Logs
Trace Messages
Log Settings
Apache VFS based file transport
Ability define more than one administration account
General UI usability enhancements and fixes
Scheduled Task support
New mediators - XQuery, POJO Command, DB Report and DB Lookup
New EIP mediators - Split/Clone and aggregator mediators
Cache, Throttle and Class mediator enhancements
Improved logging and tracing support, and reconfiguration of Log4J instance at runtime
Samples (sample 0 - sample 420)
- Message mediation samples
- Advanced mediations with endpoints
- Quality of Service addition or deduction samples in message mediation
- ESB Proxy service samples
- QoS addition and deduction for service mediation (proxy) samples
- Transport samples and switching transports
- ESB tasks
- Advanced mediations with advanced mediators (Script Mediator, DBLookup / DBReport, Throttle Mediator, Class Mediator, XQuery Mediator, Iterate / Clone, Caching)

Out of Scope

Testing on Web Browsers other than IE 7.x and Firefox 2.x
Testing on non-Linux and non-Solaris Operating Systems (However we will perform a installation and smoke tests on MS Windows)

QA test process

The summarized QA process is given in WSO2 QA Process page. This section describes the tasks which will be performed during a test cycle.

Smoke/Installation testing

The smoke test is designed to ensure that each relevant component in the build is in a state where QA testing can be carried out effectively. Detailed testing will be carried out only if the smoke test is successful. A subset of QA functional test cases which covers the critical functions are taken as the smoke test cases and will be executed with each QA build.

During the installation testing, QA team deploy and configure the WSO2 ESB on a particular environment as stated in the installation guide. All the release artifacts such as README files, release notes will be verified. Also, the release source distribution will be built into a clean machine (and maven repo) and the resulting artifacts verified with the binary release.

Installation testing will be suspended if one or more of the following is encountered;

If the installation guide does not have the necessary steps to configure and set up the product
Files needed for installation are not included in the release (eg:- startup.sh file is missing etc..)
- Installation / Smoke testing should also verify the list of Jars/third party dependencies which will be informed by the product manager before the release. There should be no extra or missing Jars or differences in versions. Must also verify any endorsed Jars etc where applicable.
- The product manager should confirm to the QA that the LICENSE and NOTICE files have been updated for the release.

UI/Usability testing

Usability testing will assess the user interfaces for ease of use, navigation and understanding. QA team verifies the pages, hyperlinks and the other UI components of the ESB admin console during UI testing.

Functional testing

Functional testing is designed to verify the functional requirements are met in the WSO2 ESB release. The ESB and Synapse 1.0 QA Testing Check List will be used as the reference for functional testing. All the test cases defined in that document will be executed and the defects are reported though JIRA.

Functional testing will be suspended if show stoppers or critical issues which prevent the testing of major functionalities are encountered.

Functional testing should accommodate the following advanced testing scenarios wherever applicable:

Testing SOAP 1.2 messages in addition to SOAP 1.1
Testing with WSDL 2.0's in addition t o WSDL 1.1's for Proxy services and WSDL endpoints etc.
Testing with large requests - i.e. in addition to MTOM, over 100K and upto 15MB (e.g. with Data services) SOAP/POX messages should be tested
Testing for re-connection after network disconnects - the JMS, DB mediators etc should be tested for automatic detection and reporting of NW failures at correct severity levels to the logs; and re-connection once resources become available.
Performance testing - need to load test SOAP, POX and REST requests as well as scheduled tasks.
The server should be kept running overnight etc where applicable - or on a server for many days and the memory consumption recoded
Clustering support - where applicable e.g. Caching/Throttling

Regression testing

Regression testing and retesting cycles are performed to ensure that;

Changes and defect fixes do not adversely impact previously released versions
There is an increased stability of the product
Bugs in the previous releases are addressed correctly

Regression testing will be suspended if showstoppers or critical issues which prevent the testing of major functionalities are encountered.

Performance testing

QA team plans to do a basic performance test cycle using Apache Bench. Soap requests will be sent to a given endpoint and measure the response time using # number of concurrent users with multiple requests.

Test Environment

Quality assurance testing will be conducted on the following software platforms.

Supported Operating Systems

Linux (Ubuntu) / Fedora or Redhat (the testa, testb, testc machines)
Solaris - at least the installation, daemon process functionality and smoke test
MS Windows - at least the installation, running as a service and a smoke test

Supported / Tested JDKs

Sun JDK 1.5.x only

Supported Browsers

Firefox 2.x
IE 7.x

Other optional dependencies or products used for testing

ActiveMQ 4.1.x
JBoss 4.2.1-GA / JDK 1.5
Derby 10.3.1.4

Defects Management

WSO2 JIRA will be used as the issue tracking system for release 1.5. The detailed defect tracking information is available at Defects Management