Example: Performance Test Plan
This test plan serves as the document that describes the lab equipment required to perform the testing and the scheduled sequence of experiments required to gather the data called out in the workload specification.
Relationships
Related Elements
Main Description

Introduction

Online Gardening Supplies is an internet-based outlet selling gardening supplies to the general public. Their web site generates 100% of their sales and is critical to the survival and success of the business. OGS is re-hosting their website on IBM Series x346 servers using a three tiered IBM middleware solution that is based on IBM WebSphere Application Server v6.1, IBM DB2 Database Server, and IBM HTTP Server v6.1.

This test plan serves as the document that describes the lab equipment required to perform the testing and the scheduled sequence of experiments required to gather the data called out in the workload specification. These tests are designed to measure the performance results in the presence of the peak commercial user load on the OGS website after it is placed into production. Performance of the OGS system under this workload will determine whether the system meets OGS's performance requirements and whether OGS will accept the system as designed.

Environment

Test Environment

The test environment should reflect the production system setup in every way possible. In this case the back-end data base is currently not the same and does not exhibit the scalability of a separate server running a production IBM DB2 database environment. Without adding this capability to the test system, no formal testing can be performed. Once the DB2 environment is in place, the system capacity, robustness, and availability testing can be performed.

System Environment

The system under test consists of a front-end IBM HTTP Server v6.1 web server hosted on a dual processor server running Windows 2003 and an IBM WebSphere Application Server v6.1 hosted on a dual processor server running Windows 2003. In the production environment, a separate server would be running the database using IBM DB2 Database Server. For these preliminary tests, the database will reside on the same server as the application server. This will prevent realistic traffic volumes from being run through this configuration. Performance bottlenecks in the J2EE application code will be explored prior to deployment of a separate database server. This database deployment will be required to get production volume traffic and response time estimates can be measured.

The performance testing complex will consist of an RPT console machine and two test agent machines that will be used to drive the system under test described above. All systems are connected over the 100MB LAN fabric used in the RPT mini-lab network.

Test Schedule

The overall test schedule consists of the following test phases:

  1. Equipment, middleware, and application setup (5 days)
  2. Performance test development (3 days)
  3. Application tuning of J2EE code (5 days)
  4. System capacity testing (2 days)
  5. Robustness testing (2 days)
  6. Availability testing (10 days)

Each of these test phases requires the completion of the previous phase. The database deployment is required for Phases 4 through 6.

Equipment and Software Setup

Installation of all system hardware and networking must be in place. The operating systems, anti-virus and firewall software then need to be installed. The web server and application server should be installed next. The RPT test agent software for data collection should then be installed on the web and application servers. The application server should be instrumented and the data collection monitors started on both servers. Finally a stable version of the application code should be deployed on the application server. Browser access to the application through the front end web server completes the sanity testing for the application environment.

The RPT console and test agent systems need to be installed with the latest product version available making sure that the test agents installed on all systems are the same version as the version of the RPT console. The RPT console server needs to be licensed for development and point to an IBM Rational license server hosting virtual tester playback license packs for use during the multi-user testing of the application.

Deliverable Artifacts
  1. Test lab machines configured with OS, middleware, and application software.
  2. RPT console and test agent machines configured with latest RPT version.

Performance Test Development

The performance test scenarios described in the OGS Workload Specification shall be captured using the RPT recorder, verification points added, and input data variation added. The tests shall be debugged for improper operation in single execution, looped execution, few user parallel execution, and then in a homogeneous stress load environment. Then these tests will be declared baselined and ready to be used in a multiple scenario workload environment. Any required datapools that need to be built should be built and tested as part of this debugging process.

Deliverable Artifacts
  1. Performance test project
  2. Fully debugged and parameterized tests for each of the test scenarios
  3. Test schedules used to stress test each individual test
  4. Test results from individual test playback and stress test playback for each test
  5. Datapool assets to support the required input data variation for each test

Application Server Tuning

First the ramp-up test is performed to identify the basic application server capacity based on response times and system resource usage. Then perform the transaction breakdown data collection experiment at two/thirds of capacity. Analyze the transaction data for any slow transactions and work with development to resolve any unexpected bottlenecks in the application. Finally perform the one hour steady state test with resource monitoring to make sure the application performance stays steady. Report any suspected memory leaks back to development for analysis.

Deliverable Artifacts
  1. Performance test schedule implementing a gradual ramp up to determine application system capacity when response times become unacceptable
  2. Performance test schedule used to tune a single application server at two/thirds server capacity with transaction breakdown tracing turned on
  3. Performance test schedule used to tune a single application server at full load with resource monitoring configured for tracking server resources
  4. Test results from a gradual ramp up past the full load capacity of the application server to identify full capacity loading
  5. Test results from a 15 minute steady state playback at two/thirds server capacity with transaction breakdown tracing turned on
  6. Test results from a 1 hour steady state playback at full load server capacity with resource monitoring to track java process size growth to verify there are no memory leaks and that performance stays steady

System Capacity Determination

On the complete system including the fully deployed and sized database server, run the ramp up schedule to determine the system capacity full load point using response times and system resource usage. Verify that the test agents can handle the full system capacity workload and take accurate measurements by verifying agent resource usage. Take the ramp up measurement well beyond the system capacity whether it is more or less than the expected production workload.

Deliverable Artifacts
  1. Performance test schedule implementing a gradual ramp up to determine system capacity when response times become unacceptable
  2. Test results with resource monitoring for all servers including the test agents

System Capacity Measurement

Run the workload at the measured system capacity for a period of at least one hour while monitoring system resources to verify that the system can sustain the capacity found during the ramp-up measurement. Verify that system resources on the test agents are within satisfactory operating region so the measurements can be validated.

Deliverable Artifacts
  1. Performance test schedule implementing the maximum system capacity workload
  2. Test results with resource monitoring for all servers including the test agents from a steady state run lasting at least 1 hour

Robustness Testing

Run the workload at 150% of system capacity for at least one hour to prove that you get consistent stable response times and throughput processing even though the response time is not acceptable. This test is to demonstrate that the system remains stable even under an overload condition.

Deliverable Artifacts
  1. Performance test schedule implementing 150% of expected production workload
  2. Test results with resource monitoring for all servers including the test agents from a steady state run lasting at least 1 hour

Availability Testing

Run the system at expected production workload for 72 hours to demonstrate long term stability of the system without requiring restarts and to show that system processing throughput and response time can be maintained at peak efficiency for long periods of time.

Deliverable Artifacts
  1. Performance test schedule implementing the expected production workload modified for a 72 hour continuous operation with reduced data collection
  2. Test results from a steady state run lasting at least 72 hours