Purpose
Monitoring the percentage of automated testing in a project helps the team identify the potential for, and progress
toward, test-case automation in order to reduce costs and effort.
Test automation reduces labor costs up to 30-40% in the first year (one-time, not counting effort to automate).
Automated testing is essential for reducing development risk, especially in iterative development. With numerous builds
or executable releases, it becomes more difficult to fully test and therefore validate quality without automated
testing.
The initial test design effort for an automated test case is typically 30-40% more than for a manual test case, since
you must code the design and validate the test code. However, the payback is in test execution, where automated tests
can decrease test times by up to 80%-90%.
Definition
Calculate the percentage of total test cases that are automated as follows:
(Total Number of Automated Test Cases / Total Number of Test Cases) * 100
Analysis
A good way to monitor Percentage of Automated Testing over time is to use a stacked bar chart which has the added
benefit of showing the growth in overall number of test cases. The percentage should increase over time.
Typical ROI case:
-
Assume the following - Time to execute manual test is T
-
Assume that - Coding an automated test is 1.3* T
-
Assume that - Automated test is 0.2 *T per instance
-
Assume that maintenance of automated test is 0.3 * T annually
-
Assume that test is run N times where N is an integer of 1 to 6
-
Thus cost of running manual test over a year is N*T
-
Thus cost of executing automated tests over a year is 1.3*T+(N*0.2T)+0.3T
The following table shows projected cost savings for a year of automated versus manual testing:
Value for N
|
Manual Test Cost
|
Automated Test Cost
|
1
|
n
|
1.8n
|
2
|
2n
|
2n
|
3
|
3n
|
2.2n
|
4
|
4n
|
2.4n
|
5
|
5n
|
2.6n
|
6
|
6n
|
2.8n
|
Based on the above, the break-even point is about two test iterations per year. If you run the manual test more than
twice a year, you can see a significant return of about 30% labor cost for three iterations, and 55% for six
iterations. Note that this does not include maintenance estimates.
Frequency and reporting
Data is collected and reviewed at the end of each iteration.
Collection and reporting tools
Test case data is captured in IBM® Rational® Quality Manager®. IBM® Rational® Functional Tester® and IBM® Rational®
Performance Tester® both capture test case automation data.
Pitfalls, advice, and countermeasures
-
There is some effort required to maintain automated tests (e.g. updating scripts). Test code should be placed under
source control just like development code. This often requires a dedicated centralized team that can manage the
code and provide core expertise to projects.
-
Automate most-frequently-run test cases first (see theTest-Case Defect Yield metric). You may need to separate out
test cases with high level of graphical user interface change or other inhibitors to automation.
-
Percent of automated test coverage is not an indicator of test effectiveness.
-
Automated testing is not suitable for all projects. In some cases (such as in very small projects) the time to
create and maintain an automated test script is greater than the time required to create a manual test.
|