Guideline: Workload Model
This guideline describes the main aspects that you need to consider during the development of a Workload Model.
Relationships
Main Description

Workload Model of the System's Busy Hour

The workload model usually consists of a characterization of how much work the system must perform at its busiest time. This represents the peak loaded capacity that the system must support while maintaining satisfactory user visible characteristics such as responsiveness to user requests for information. The derivation of the workload model is an important skill expected of a senior systems analyst, a system architect, a business analyst or a performance testing specialist. However, without a clear analytical workload model, the performance testing project can not truly get started. Often the performance tester is expected to bring together the business analyst and system architect and build a workload model that describes the busy hour (or hours) of the developed system.

The workload model may be derived from the number of business transactions expected during the month-end or quarter-end rush. It might also be based on the late afternoon batch report run interval that overlays an otherwise uniform transactional workload. It could also be at lunch hour on the Friday after the Thanksgiving holiday for a seasonal e-commerce Web site.

In most cases, you can think of a workload model as a matrix of transactions (or business scenarios) versus frequency of execution spread across the number of users accessing the system simultaneously. There are many forms used to describe this quantity of work. One example is shown below, where the simplified version of the system's load is reduced to two types of user transactions and two batch reports that are run in the busy hour.



Example of a Workload Model
Business Transaction System-wide Total/Hour Simultaneous Users Average Think Time [seconds] Records Impacted

Browse Item

50,000

5,000 3.00 1
Buy Items 500 100 5.00 3-5
Inventory Report 1 1 0 200,000
Sales Report 1 1 0 8,000


The impact on the system across all four components is measured and analyzed. Depending on the system design, any of the four components could cause significant interactive or batch reporting degradation beyond the minimum acceptable level of performance expected.

Test Success Criteria

Ideally the workload model will describe-with a fairly (80-90%) accurate mixture of the transactions (or business workflows)-the expected peak production workload on the system once it is in use by the business. By liberal application of the 80/20 rule, the workload model should contain the 20% of the transactions that make up 80% of the system capacity, and contain most if not all of the crucial business transactions.

Measurement of the throughput of the required transactions is likely to be a minimum standard that must be met by the system. For example, if during the peak Christmas rush, an online ordering system must process 500 shipments per hour to meet the projected increase in order volume, based on the previous three years statistics, then the new system must support this transaction rate.

On the other hand, if the submit order and order completion screens each take more than 5 seconds to appear, then most customers will abandon their shopping cart without purchasing, cancel their orders, or not return for subsequent purchases. These are more customer- oriented requirements that in fact dictate the success of the online shopping site system deployment.

Both types of requirements must be measured and successfully achieved or there is a significant business risk in going ahead with system deployment. By helping the system development team focus on these metrics and the underlying causes of non-compliant transactions or response times, the performance tester can drive a successful outcome of the performance testing project.

User Scenarios in the Workload Model

Business workflows are also considered to be the user scenarios of the system when viewed from the user's perspective. By walking through the typical user actions that make up each of these business workflows, you can create a user scenario (also known as a use case in the software requirements world) complete with typical user inputs such as login, password, product selection(s) for a commerce site or an account number, transaction type, and amount for a financial site.

The number of user scenarios that make up the workload model is very subjective and can have a major impact on how long the performance testing project takes. Simplifying the user scenarios and reducing their number have many benefits. Many times a revision of the application requires that all test scripts be recaptured. This can occur because the workflow changes in the sequencing of user screens, user input data, or even invisible underlying application parameter data passed back and forth between browser and server.

What may seem like a trivial application change from a user flow perspective may cause the test scripts to stop working properly. If you have only ten user scenarios and a highly productive tool such as Performance Tester, an application version change can be accommodated in a half day of recapture, minor editing for input data variation, and test debugging. If you have 20 or more user scenarios, this same event could cost the project a day or more of delay.

Also the number of alternate flows, conditional execution, and more extensive data variation adds to the overall time to build and maintain the set of tests that implement the workload model. Keeping the user scenarios basic and central to the important business workflows of the system is an important concept to obtain valuable, timely performance results for a reasonable project cost. Once you accept the need to abstract the workload definition to a simplified version of the true user behavior (with virtually equivalent system performance), your performance testing projects will proceed more quickly and cost effectively.

Input Data Variation in User Scenarios

Some user inputs, if varied on a per user or per iteration basis, will have an impact on system performance while other user inputs do not. By reviewing the user scenarios with the system architect, the tester can validate the selection of which user inputs to vary. By varying the data values in the user scenarios that represent the independent keys of accessed database tables, representative data retrieval times can be expected. The tester can reduce the possibility of getting optimistic test results due to database or other subsystem caching effects.

In some cases like the browsing of an online commerce site, repeatedly selecting the same five items to view in the online catalog may result in the Web server caching images and even product details. However by adding random selection of the product viewed, the system will experience a much heavier load on the non-Web server subsystems such as an image server or a backend database containing product details. This difference in the workload may impact order processing times as well as provide a more realistic estimate on the time to view a product detail page. This may be the critical customer requirement that determines the success of the Web site.

Workload Model for each Experiment

In the performance test planning for the project, the tester designs a series of experiments (or test executions) that taken together provides a methodical approach to developing the necessary test assets, making necessary test measurements, permitting the required subsystem and system tuning, making conservative estimates for application changes and test re-runs, and performing the final measurements and their analysis.

Often there is a requirement for building a subset of the full workload model to provide a means of tuning one or more of the subsystems, such as tuning the capacity of a single application server normally used in a clustered environment with a Web server front end. This can be accomplished by only having a single application server active in the cluster or by redirecting the test connections directly to the application server (bypassing the Web server and load balancer).

Problems like heap fragmentation, memory leaks, and algorithmic inefficiencies can be more easily be addressed in this smaller, less complex environment. Tuning the application server including how many instances to run on a hardware server can be done prior to scaling the system up for a full load test. Some subsystem tuning and testing involves several subsystems and have to exercised before the full load tests. One example of this is the database connection pooling to the backend database. By exercising a heavy load of database intensive operations, possibly without including the entire load of non-database related workload, the database connections from the application servers to the database can be sized properly.

As each of these experiments are defined, an associated workload definition should be listed for that experiment. By pulling these together as part of the test plan, the entire testing project team can understand when the tests and workload definitions are going to be used and what measurements and data collection needs are expected for each experiment.