Workload Model of the System's Busy Hour
The workload model usually consists of a characterization of how much work the system must perform at its busiest time.
This represents the peak loaded capacity that the system must support while maintaining satisfactory user visible
characteristics such as responsiveness to user requests for information. The derivation of the workload model is an
important skill expected of a senior systems analyst, a system architect, a business analyst or a performance testing
specialist. However, without a clear analytical workload model, the performance testing project can not truly get
started. Often the performance tester is expected to bring together the business analyst and system architect and build
a workload model that describes the busy hour (or hours) of the developed system.
The workload model may be derived from the number of business transactions expected during the month-end or quarter-end
rush. It might also be based on the late afternoon batch report run interval that overlays an otherwise uniform
transactional workload. It could also be at lunch hour on the Friday after the Thanksgiving holiday for a seasonal
e-commerce Web site.
In most cases, you can think of a workload model as a matrix of transactions (or business scenarios) versus frequency
of execution spread across the number of users accessing the system simultaneously. There are many forms used to
describe this quantity of work. One example is shown below, where the simplified version of the system's load is
reduced to two types of user transactions and two batch reports that are run in the busy hour.
Example of a Workload Model
Business Transaction
|
System-wide Total/Hour
|
Simultaneous Users
|
Average Think Time [seconds]
|
Records Impacted
|
Browse Item
|
50,000
|
5,000
|
3.00
|
1
|
Buy Items
|
500
|
100
|
5.00
|
3-5
|
Inventory Report
|
1
|
1
|
0
|
200,000
|
Sales Report
|
1
|
1
|
0
|
8,000
|
The impact on the system across all four components is measured and analyzed. Depending on the system design, any of
the four components could cause significant interactive or batch reporting degradation beyond the minimum acceptable
level of performance expected.
Test Success Criteria
Ideally the workload model will describe-with a fairly (80-90%) accurate mixture of the transactions (or business
workflows)-the expected peak production workload on the system once it is in use by the business. By liberal
application of the 80/20 rule, the workload model should contain the 20% of the transactions that make up 80% of the
system capacity, and contain most if not all of the crucial business transactions.
Measurement of the throughput of the required transactions is likely to be a minimum standard that must be met by the
system. For example, if during the peak Christmas rush, an online ordering system must process 500 shipments per hour
to meet the projected increase in order volume, based on the previous three years statistics, then the new system must
support this transaction rate.
On the other hand, if the submit order and order completion screens each take more than 5 seconds to appear, then most
customers will abandon their shopping cart without purchasing, cancel their orders, or not return for subsequent
purchases. These are more customer- oriented requirements that in fact dictate the success of the online shopping site
system deployment.
Both types of requirements must be measured and successfully achieved or there is a significant business risk in going
ahead with system deployment. By helping the system development team focus on these metrics and the underlying causes
of non-compliant transactions or response times, the performance tester can drive a successful outcome of the
performance testing project.
User Scenarios in the Workload Model
Business workflows are also considered to be the user scenarios of the system when viewed from the user's perspective.
By walking through the typical user actions that make up each of these business workflows, you can create a user
scenario (also known as a use case in the software requirements world) complete with typical user inputs such as login,
password, product selection(s) for a commerce site or an account number, transaction type, and amount for a financial
site.
The number of user scenarios that make up the workload model is very subjective and can have a major impact on how long
the performance testing project takes. Simplifying the user scenarios and reducing their number have many benefits.
Many times a revision of the application requires that all test scripts be recaptured. This can occur because the
workflow changes in the sequencing of user screens, user input data, or even invisible underlying application parameter
data passed back and forth between browser and server.
What may seem like a trivial application change from a user flow perspective may cause the test scripts to stop working
properly. If you have only ten user scenarios and a highly productive tool such as Performance Tester, an application
version change can be accommodated in a half day of recapture, minor editing for input data variation, and test
debugging. If you have 20 or more user scenarios, this same event could cost the project a day or more of delay.
Also the number of alternate flows, conditional execution, and more extensive data variation adds to the overall time
to build and maintain the set of tests that implement the workload model. Keeping the user scenarios basic and central
to the important business workflows of the system is an important concept to obtain valuable, timely performance
results for a reasonable project cost. Once you accept the need to abstract the workload definition to a simplified
version of the true user behavior (with virtually equivalent system performance), your performance testing projects
will proceed more quickly and cost effectively.
Input Data Variation in User Scenarios
Some user inputs, if varied on a per user or per iteration basis, will have an impact on system performance while other
user inputs do not. By reviewing the user scenarios with the system architect, the tester can validate the selection of
which user inputs to vary. By varying the data values in the user scenarios that represent the independent keys of
accessed database tables, representative data retrieval times can be expected. The tester can reduce the possibility of
getting optimistic test results due to database or other subsystem caching effects.
In some cases like the browsing of an online commerce site, repeatedly selecting the same five items to view in the
online catalog may result in the Web server caching images and even product details. However by adding random selection
of the product viewed, the system will experience a much heavier load on the non-Web server subsystems such as an image
server or a backend database containing product details. This difference in the workload may impact order processing
times as well as provide a more realistic estimate on the time to view a product detail page. This may be the critical
customer requirement that determines the success of the Web site.
Workload Model for each Experiment
In the performance test planning for the project, the tester designs a series of experiments (or test executions) that
taken together provides a methodical approach to developing the necessary test assets, making necessary test
measurements, permitting the required subsystem and system tuning, making conservative estimates for application
changes and test re-runs, and performing the final measurements and their analysis.
Often there is a requirement for building a subset of the full workload model to provide a means of tuning one or more
of the subsystems, such as tuning the capacity of a single application server normally used in a clustered environment
with a Web server front end. This can be accomplished by only having a single application server active in the cluster
or by redirecting the test connections directly to the application server (bypassing the Web server and load balancer).
Problems like heap fragmentation, memory leaks, and algorithmic inefficiencies can be more easily be addressed in this
smaller, less complex environment. Tuning the application server including how many instances to run on a hardware
server can be done prior to scaling the system up for a full load test. Some subsystem tuning and testing involves
several subsystems and have to exercised before the full load tests. One example of this is the database connection
pooling to the backend database. By exercising a heavy load of database intensive operations, possibly without
including the entire load of non-database related workload, the database connections from the application servers to
the database can be sized properly.
As each of these experiments are defined, an associated workload definition should be listed for that experiment. By
pulling these together as part of the test plan, the entire testing project team can understand when the tests and
workload definitions are going to be used and what measurements and data collection needs are expected for each
experiment.
|