Test Execution Status
This metric tracks the number of tests planned, implemented, attempted, passed, failed, and blocked.
Main Description

Purpose

This metric helps a team understand overall solution quality by studying two factors: test completeness and test success. The test completion rate enables the team to manage what part of the test effort remains, including the risk associated with potentially undiscovered quality issues. The test success rate helps the team to decide whether the software or system is working correctly. This metric should be used in conjunction with the Defect Trends metric to determine release readiness of the solution.

When the term "test" is used in this metric, it refers to the executable tests that contain the steps needed to run against the actual solution under development. These tests can be manual or automated. The Rational Unified Process refers to these tests as "test scripts", but in some test methodologies and test tools, these are referred to as test cases. To avoid confusion, this metric uses the term "test."

Definition

Test Execution Status is measured by reporting the following items once per iteration and trending throughout the release cycle:

  • Tests planned.
  • Tests implemented.
  • Tests attempted.
  • Passed tests.
  • Failed tests.
  • Blocked tests.


The term iteration is sometimes called "test cycle," but these are not synonymous. A single iteration may have multiple test cycles, based on the build cycle. The recommendation is to capture this metric once per iteration, regardless of the number of test cycles.

Terms

Tests planned = The number of tests scheduled to be executed in the iteration.

Tests implemented = The number of tests built and ready to be executed - both manually and automatically - in the iteration.

Tests attempted = The number of tests that have been executed, and is the sum of the passed, failed and blocked tests.

Passed tests = The number of tests that have a most recent result of pass.

Failed tests = The number of tests that have a most recent result of failed.

Blocked tests = The number of tests that cannot be executed completely to the last step of the test. For manual tests, this means the tester could not execute all the steps of the test. For automated tests, the automated testing tool reports a passing result, but the human test analyst determines that the test was invalid using information outside the scope of what the automated testing tool can report.

Build Health is captured in IBM® Rational® Team Concert® and BM® Rational® Insight®.

Analysis

Use either a line or bar graph that shows number of tests on the y-axis and the iterations along the x-axis. Categorize the tests as indicated above. Ideally, show the results against a planned schedule for implementation and successful execution. The following patterns might occur:

Rising slope

  • Tests planned, tests implemented, tests attempted, and passed tests: This is the desired trend.
  • Failed tests: Indicates decreasing solution quality and/or decreasing requirement quality.
  • Blocked tests: Indicates the test effort may be falling behind schedule and the quality of the solution is more unknown. Also, may be an indicator of technical problems or test data problems in the test environment.


Falling slope

  • Tests planned: Indicates tests are being removed from the scope of the test effort, possibly indicating a decrease in overall scope for the release.
  • Tests implemented: Indicates there are not enough test resources to write the planned tests.
  • Tests attempted: Indicates there are not enough test resources to execute the planned and implemented tests.
  • Passed tests: Depending on the trends of planned, implemented and attempted tests, this pattern usually indicates decreasing solution quality, and/or previously passing tests are now failing.
  • Failed tests and blocked tests: This is the desired trend.


Flat line

  • Tests planned, tests implemented, and tests attempted: Indicates new tests are not being added to the overall test effort for the release. Some root causes are: lack of test resources to implement and/or execute tests, lack of clear requirements, no new requirements being delivered to test.
  • Passed tests: Indicates defects are not being corrected. Could also indicate a coincidental net zero difference in the number of passing tests.
  • Failed tests and blocked tests: Indicates there is a lack of test resources to execute previously failed or blocked tests, or defects are not being corrected, or a coincidental net zero difference in the number of failing or blocked tests, or some combination of these issues. The test schedule may be in jeopardy in this scenario.

Hockey stick

A hockey stick trend gradually increases or decreases, then takes a sharp turn in the upward direction, typically late in the release cycle. This means the project is experiencing surprises at a time when things should be routine.

  • Tests planned, tests implemented, tests attempted: Indicates many new tests are being added to the test effort, possibly due to: new requirements added to the scope of the project, previously ambiguous requirements have been clarified, additional test resources added.
  • Passed tests: Indicates the following possible scenarios: defects are being corrected and verified more quickly, requirements are more clear, additional test resources have been added, or some combination of these items.
  • Failed tests and blocked tests: Indicates the following possible issues: new tests are failing or blocked, previously passing tests are now failing or blocked, requirements are not clear, previously corrected defects are recurring, new defects are being discovered.

Reverse hockey stick

A reverse hockey stick trend gradually increases or decreases, then takes a sharp turn in the downward direction, typically late in the release cycle. This means the project is experiencing surprises at a time when things should be routine.

  • Tests planned, tests implemented, tests attempted: Indicates tests have been removed from the scope of the project - perhaps due to removal of requirements.
  • Passed tests: Indicates that a large number of tests that were previously passing are suddenly no longer passing. Possible sources for this issue are: previously passing tests are now failing and/or newly delivered solution quality is decreasing.
  • Failed tests and blocked tests: Indicates the following possible scenarios: defects are being corrected and verified more quickly, requirements are more clear, additional test resources have been added, or some combination of these items.


Perfect slope

Perfect slope describes a trend where the actual data exactly matches the expected behavior for a metric. Sometimes there are patterns in perfect slope situations that allow us to give specific advice on corrective action. For this metric, other than investigating potential bad data, there is no specific advice.


The following graph shows an example of two of the test execution status report trends - passed tests and failed tests.

Test Execution Status

Frequency and reporting

Data is captured each day and monitored at the end of each iteration to help identify trends.

Collection and reporting tools

Test execution data is captured in IBM® Rational® Quality Manager®. IBM® Rational® Insight® provides an out of the box report focused on implemented tests and their status.

Assumptions and prerequisites

  • Test plans are updated as part of iteration planning and compared to actuals at the end of each iteration.
  • Test execution data is captured in a tool, tracking the status of each attempted test.

Pitfalls, advice, and countermeasures for this metric

The following items are indicators of measurement pitfalls and should be used to corroborate this metric:

  • Release size
  • Requirement quality
  • Test capacity
  • Defect aging