Purpose
Using this metric, a team can establish links between defects and specific process activities and their outputs (e.g.
architecture, requirements, design, code, test). When the source of the defect is known, a fix can be implemented that
has the potential to prevent future defects associated with that source. Capturing defects early in the lifecycle is a
means of defect prevention for later phases of the lifecycle when correction is more expensive.
Analyzing Defect Density by Origin trends helps a team identify process improvements for each activity that can prevent
the injection of similar defects in the future. This can help teams and organizations improve in areas such as cost,
quality, and time to value.
Definition
In order to associate defects with their source of origin it is necessary to implement a classification system. IBM's Orthogonal Defect
Classification is a framework for categorizing each defect by mutually exclusive attributes. These attributes are
used to establish patterns helpful for root cause analysis. The Target attribute represents the high level identity of
the entity that was fixed. This is considered to be the source of the defect (e.g. architecture, design, code,
requirement)
Count: Number of identified defects, grouped by source
Analysis
Use a bar chart or grouped bar chart to show the number of defects associated with each source. A stacked bar chart can
be used to additionally monitor by status. To show trends over time, use a series of bars, or a line chart. Plot by
iteration or by phase.
Defect source distribution varies over time, and provides an indication of where development stands in the lifecycle.
Expected trend - Ideally, most architecture defects are identified in very early iterations, tapering
off to very few in the last half of the lifecycle with none identified at the end. Requirements defects also peak early
in the lifecycle tapering off with a few peaks and valleys to the end. Design defects peak a bit later than
architecture but taper off at a slower rate through the second half of the lifecycle. Code and test defects peak later
in the process.
Monitor changes in the distribution of defects throughout the lifecycle. Set targets based on development approach and
the level of uncertainty in the project. Monitor unexpected peaks to identify issues with specific artifacts or
activities, and analyze for potential process improvements.
For example:
-
If a high number of requirements-based defects are found late in the lifecycle, the team should assess their
requirements elicitation and traceability processes and tools. If a high number of defects are traced to a specific
requirement, there are likely quality issues with that requirement that need to be addressed.
-
When a high number of architecture defects are identified, the team should perform sufficient analysis to prevent
costlier defects associated with the architecture from arising late in the lifecycle.
Frequency and reporting
Defect classification should begin in the first iteration, and results reviewed by the team in each iteration
assessment. Stakeholders should review the data as each phase milestone is reached. The data is also reviewed at the
end of each release as an input to planning for the next release.
Trend reports from projects across the organization can be rolled up to the organization level and tracked over time as
process improvements are introduced.
Collection and reporting tools
IBM® Rational® ClearQuest® collects Defect Density by Origin data. It is necessary to implement custom attributes to
support the Orthogonal Defect Classification framework. IBM® Rational® Quality Manager® is also used to collect data
for this metric.
Assumptions and prerequisites
-
A semantic classification of defects is established and well understood by the team.
-
Upon closure, each defect is assigned a Target value to associate it with its source
-
The team has implemented effective test processes with sufficient coverage to identify defects associated with all
sources
Pitfalls, advice, and countermeasures
-
Team members must agree on the source of each defect. As each defect is classified, there must be a clear
indication of something missing or incorrect in a specific artifact/ source.
-
Defect classification provides fast feedback to the team, allowing them to address quality issues with very
specific artifacts and activities early in the process.
-
It is essential for results to feed back into the process in order to drive improvement. Without this key step,
other projects in the organization cannot learn from previous defect patterns and take action to prevent the same
mistakes happening in the future. Use the metric to monitor process improvements as they are introduced.
-
A lack of consistency and discipline in classifying defects can lead to skewed results, making trend analysis
useless.
|