Purpose
Defect Removal Efficiency is a key indicator of process effectiveness as well as quality. It is a measure of the
percentage of defects eliminated in various phases and activities of the lifecycle during development and after the
software is released. An effective defect removal process will limit the number of latent defects, and increase
customer satisfaction.
Definition
Count:
-
the number of defects found, by severity, in each test cycle (unit testing, system testing, etc) or phase
-
the number of defects removed, by severity, in each test cycle or phase
Calculate the Defect Removal Efficiency for each severity in each test cycle as follows:
Defect Removal Efficiency = (Total Defects Removed / Total Defects Found) * 100
Defect Removal Efficiency scope can be expanded to include defects found in formal design and code reviews, and other
project activities.
Analysis
Use a set of trend lines to monitor Defect Removal Efficiency (one for each test cycle, activity, or phase measured).
Plot Defect Removal Efficiency percentage on the Y axis and test cycles/ phase/ activity on the X axis.
Expected trend - The percentage of defects found and removed in development compared to after the
product ships is very high. Most teams set a target of 85-95%. Monitor the effectiveness for each activity/ phase to
determine where the team is most and least effective in finding and removing defects.
For Agile and/ or iterative development approaches, testing begins early in the development lifecycle.
-
An upward trend of defects found (cumulative) from the beginning of the lifecycle is expected
-
The number of defects found (in each iteration) should be consistent across the lifecycle. Expect a reduction
in defects found at the end of lifecycle (if testing has been performed properly)
-
The number of closed defects should be equal or very close to the number of defects found in that
iteration (i.e. the opened defects should burn down to zero)
For waterfall or non-iterative development projects, the test cycle usually begins late in the development lifecycle.
-
The number of defects found trends upward toward the middle to the end of the development lifecycle. The
defect discovery rate slows down at the end of development. Expect a bell curve for defect arrival.
-
The number of opened defects trends downward to zero when development is coming to an end.
Low Defect Removal Efficiency - When the Defect Removal Efficiency percentage is low, it is an
indication of poor processes. Confirm that the team has implemented effective practices and tools to identify defects
in each phase of development (and throughout when following an iterative approach). For example, if the
percentage of defect removal is low during testing activities, the team may need to improve the quality of their test
cases or their test coverage of requirements. If reviews aren't effective in the defect removal process, examine your
review processes to see why defects are missed.
The potential for increasing the percentage increases dramatically when a combination of unit and system testing, code
and design reviews are implemented. The team should balance the benefit of implementing these practices with the
overhead based on the complexity and scale of the project.
For an Agile development approach, the majority of defects are expected to be removed by developer
test and integration test (regression test/ build test). Performing functional test or system test should yield a low
number of defects.
Frequency and reporting
Data is collected and reported at the end of each test cycle and phase. Defect Removal Efficiency targets are set by
the team.
Collection and reporting tools
IBM® Rational® Quality Manager® collects Defect Removal Efficiency Data.
Assumptions and prerequisites
-
Each identified defect is counted in the test cycle/ phase/ activity in which it was found
-
Each removed defect is counted in the test cycle/ phase/ activity in which it was removed
Pitfalls, advice, and countermeasures
-
High Defect Removal Efficiency is an indicator of customer satisfaction. Fewer defects post-ship typically
correlates to high customer satisfaction levels.
-
While defect discovery and removal is important, defect prevention is much more efficient and effective in reducing
defects.
|