Introduction
This document defines the strategy and plan for testing the WSO2 WSAS 2.0 release. All QA testing activities during the release process will be based on this document.
Scope and Test Coverage
WSO2 WSAS is an integrated Web services Platform which offers a complete middleware solution. It is a lightweight, high performing platform for Service Oriented Architectures, enabling business logic and applications. Bringing together a number Apache Web services projects, WSO2 WSAS provides a secure, transactional and reliable runtime for deploying and managing Web services.
The following test types will be carried out during the QA testing cycle of WSAS 2.0 release.
- Installation/Smoke testing
- UI/Usability testing
- Functional testing
- Regression testing
Features list of the product for release 2.0
The new features which will be available in the release 2.0 are given in Feature list for future WSO2 WSAS releases Wiki page. In addition to that the following list of features will be tested.
- Services and service group management
- Modules management
- User Management
- Transports
- Security/Key stores
- Logging management
- System monitoring
- System
- Statistics
- Tracer
- Logs
- Flows
- Samples
- CommodityQuote
- Trader
- Chad
- Javascript Service
- TinyURL
- Tools
- TCPMon
- WSDL2Java
- JAVA2Wsdl
- Change Password
- MSI/Deb/rpm installer testing
- Eclipse plugin integration
- Testing on different databases - MySQL, Derby
Out of Scope
Unit and Integration testing will not be carried out during QA testing.
QA test process
The summarized QA process is given in WSO2 QA Process page. This section describes the tasks which will be performed during a test cycle.
Smoke/Installation testing
The smoke test is designed to ensure that each relevant component in the build is in a state where QA testing can be carried out effectively. Detailed testing will be carried out only if the smoke test is successful. A subset of QA functional test cases which covers the critical functions are taken as the smoke test cases and will be executed with each QA build.
During the installation testing, QA team deploy and configure the WSO2 WebServices Application Server on a particular environment as stated in the installation guide. All the release artifacts such as README files, release notes will be verified. Also, the release will be checked out via SVN into a clean machine and build the product according to the installation guide.
Installation testing will be suspended if one or more of the following is encountered;
- If the installation guide does not have the necessary steps to configure and set up the product
- Files needed for installation are not included in the release (eg:- startup.sh file is missing etc..)
UI/Usability testing
Usability testing will assess the user interfaces for ease of use, navigation and understanding. QA team verifies the pages, hyperlinks and the other UI components of the WSAS admin console during UI testing.
Functional testing
Functional testing is designed to verify the functional requirements are met in the WSO2 WSAS release. The WSO2 WSAS QA testing checklist will be used as the reference for functional testing. All the test cases defined in that document will be executed and the defects are reported though JIRA.
Functional testing will be suspended if showstoppers or critical issues which prevent the testing of major functionalities are encountered.
Regression testing
Regression testing and retesting cycles are performed to ensure that;
- Changes and defect fixes do not adversely impact previously released versions
- There is an increased stability of the product
- Bugs in the previous releases are addressed correctly
Regression testing will be suspended if showstoppers or critical issues which prevent the testing of major functionalities are encountered.
Performance testing
QA team plans to do a basic performance test cycle using Apache Bench. Soap requests will be sent to a given endpoint and measure the response time using # number of concurrent users with multiple requests.
Test Environment
Quality assurance testing will be conducted on the following software platforms.
- JBoss 4.0.x, 3.2.x/jdk1.4/jdk1.5/jdk1.6
- gerenimo-jetty-j2ee-1.1.*/jdk1.4/jdk1.5/jdk1.6
- Weblogic 9.2/jdk1.5/jdk1.6
- All of the above configurations will be tested on both ubuntu linux and WinXP operating systems.
Defects Management
WSO2 JIRA will be used as the issue tracking system for release 2.0. The detailed defect tracking information is available at Defects Management
QA Testing Schedule
The draft QA testing schedule for WSAS 2.0 release is given below.
Test preparation and design - 7th May to 20th May ---> Clustering environment configuration and updating testing check list
Testing Cycle 1 - 31st May to 06th June (RC-1) --> Deployment and Functional testing
Testing Cycle 2 - 06th June to 13th June (RC-2) --> Regression testing
Testing Cycle 3 - 13th June to 20th June (RC-3) -->Regression testing
Testing Cycle 4 - 20th June to 27th June (RC-4) --> Regression testing
Testing Cycle 5 - 27th June to 29th June (RC-5) --> Regression testing
Appendix
During the wsas 2.0 release, QA testing will be focused much on the new features stated in the 'feature list for future wsas releases'. Among those features, Clustering implementation will be required through testing to ensure proper implementation. QA team will adhere to the following approach in order to perform testing on wsas clustering .
- Identify test scenarios and update wsas QA testing checklist with them - In progress
- Configure the clustering environment with 2 nodes to identify more test scenarios - Start on 7th of May
- Extend the cluster by adding multiple nodes (This can be achieved by setting up the environment in an independent server lab)
- Configure multiple clusters and perform testing
