Defect Removal Efficiency
This metric tracks the ratio of defects discovered during development to total defects (which includes those defects found post-ship).
Main Description

Purpose

Defect Removal Efficiency is a key indicator of process effectiveness as well as quality. It is a measure of the percentage of defects eliminated in various phases and activities of the lifecycle during development and after the software is released. An effective defect removal process will limit the number of latent defects, and increase customer satisfaction.

Definition

Count:

  • the number of defects found, by severity, in each test cycle (unit testing, system testing, etc) or phase
  • the number of defects removed, by severity, in each test cycle or phase

Calculate the Defect Removal Efficiency for each severity in each test cycle as follows:

Defect Removal Efficiency = (Total Defects Removed / Total Defects Found) * 100

Defect Removal Efficiency scope can be expanded to include defects found in formal design and code reviews, and other project activities.

Analysis

Use a set of trend lines to monitor Defect Removal Efficiency (one for each test cycle, activity, or phase measured). Plot Defect Removal Efficiency percentage on the Y axis and test cycles/ phase/ activity on the X axis.

Expected trend - The percentage of defects found and removed in development compared to after the product ships is very high. Most teams set a target of 85-95%. Monitor the effectiveness for each activity/ phase to determine where the team is most and least effective in finding and removing defects.

For Agile and/ or iterative development approaches, testing begins early in the development lifecycle.

  • An upward trend of defects found (cumulative) from the beginning of the lifecycle is expected
  • The number of defects found (in each iteration) should be consistent across the lifecycle. Expect a reduction in defects found at the end of lifecycle (if testing has been performed properly)
  • The number of closed defects should be equal or very close to the number of defects found in that iteration (i.e. the opened defects should burn down to zero)

For waterfall or non-iterative development projects, the test cycle usually begins late in the development lifecycle.

  • The number of defects found trends upward toward the middle to the end of the development lifecycle. The defect discovery rate slows down at the end of development. Expect a bell curve for defect arrival.
  • The number of opened defects trends downward to zero when development is coming to an end.

Low Defect Removal Efficiency - When the Defect Removal Efficiency percentage is low, it is an indication of poor processes. Confirm that the team has implemented effective practices and tools to identify defects in each phase of development (and throughout when following an iterative approach). For example, if the percentage of defect removal is low during testing activities, the team may need to improve the quality of their test cases or their test coverage of requirements. If reviews aren't effective in the defect removal process, examine your review processes to see why defects are missed.

The potential for increasing the percentage increases dramatically when a combination of unit and system testing, code and design reviews are implemented. The team should balance the benefit of implementing these practices with the overhead based on the complexity and scale of the project.

For an Agile development approach, the majority of defects are expected to be removed by developer test and integration test (regression test/ build test). Performing functional test or system test should yield a low number of defects.

Frequency and reporting

Data is collected and reported at the end of each test cycle and phase. Defect Removal Efficiency targets are set by the team.

Collection and reporting tools

IBM® Rational® Quality Manager® collects Defect Removal Efficiency Data.

Assumptions and prerequisites

  • Each identified defect is counted in the test cycle/ phase/ activity in which it was found
  • Each removed defect is counted in the test cycle/ phase/ activity in which it was removed

Pitfalls, advice, and countermeasures

  • High Defect Removal Efficiency is an indicator of customer satisfaction. Fewer defects post-ship typically correlates to high customer satisfaction levels.
  • While defect discovery and removal is important, defect prevention is much more efficient and effective in reducing defects.