Percentage of Automated Testing
This metric tracks the percentage of testing that is automated versus manual.
Main Description

Purpose

Monitoring the percentage of automated testing in a project helps the team identify the potential for, and progress toward, test-case automation in order to reduce costs and effort.

Test automation reduces labor costs up to 30-40% in the first year (one-time, not counting effort to automate). Automated testing is essential for reducing development risk, especially in iterative development. With numerous builds or executable releases, it becomes more difficult to fully test and therefore validate quality without automated testing.

The initial test design effort for an automated test case is typically 30-40% more than for a manual test case, since you must code the design and validate the test code. However, the payback is in test execution, where automated tests can decrease test times by up to 80%-90%.

Definition

Calculate the percentage of total test cases that are automated as follows:

(Total Number of Automated Test Cases / Total Number of Test Cases) * 100


Analysis

A good way to monitor Percentage of Automated Testing over time is to use a stacked bar chart which has the added benefit of showing the growth in overall number of test cases. The percentage should increase over time.

Typical ROI case:

  • Assume the following - Time to execute manual test is T
  • Assume that - Coding an automated test is 1.3* T
  • Assume that - Automated test is 0.2 *T per instance
  • Assume that maintenance of automated test is 0.3 * T annually
  • Assume that test is run N times where N is an integer of 1 to 6
  • Thus cost of running manual test over a year is N*T
  • Thus cost of executing automated tests over a year is 1.3*T+(N*0.2T)+0.3T


The following table shows projected cost savings for a year of automated versus manual testing:



Value for N Manual Test Cost Automated Test Cost
1 n 1.8n
2 2n 2n
3 3n 2.2n
4 4n 2.4n
5 5n 2.6n
6 6n 2.8n


Based on the above, the break-even point is about two test iterations per year. If you run the manual test more than twice a year, you can see a significant return of about 30% labor cost for three iterations, and 55% for six iterations. Note that this does not include maintenance estimates.

Frequency and reporting

Data is collected and reviewed at the end of each iteration.

Collection and reporting tools

Test case data is captured in IBM® Rational® Quality Manager®. IBM® Rational® Functional Tester® and IBM® Rational® Performance Tester® both capture test case automation data.

Pitfalls, advice, and countermeasures

  • There is some effort required to maintain automated tests (e.g. updating scripts). Test code should be placed under source control just like development code. This often requires a dedicated centralized team that can manage the code and provide core expertise to projects.
  • Automate most-frequently-run test cases first (see theTest-Case Defect Yield metric). You may need to separate out test cases with high level of graphical user interface change or other inhibitors to automation.
  • Percent of automated test coverage is not an indicator of test effectiveness.
  • Automated testing is not suitable for all projects. In some cases (such as in very small projects) the time to create and maintain an automated test script is greater than the time required to create a manual test.