Defect Density by Origin
This metric tracks the number of defects associated with activities of the different stages of development.
Main Description

Purpose

Using this metric, a team can establish links between defects and specific process activities and their outputs (e.g. architecture, requirements, design, code, test). When the source of the defect is known, a fix can be implemented that has the potential to prevent future defects associated with that source. Capturing defects early in the lifecycle is a means of defect prevention for later phases of the lifecycle when correction is more expensive.

Analyzing Defect Density by Origin trends helps a team identify process improvements for each activity that can prevent the injection of similar defects in the future. This can help teams and organizations improve in areas such as cost, quality, and time to value.

Definition

In order to associate defects with their source of origin it is necessary to implement a classification system. IBM's Orthogonal Defect Classification is a framework for categorizing each defect by mutually exclusive attributes. These attributes are used to establish patterns helpful for root cause analysis. The Target attribute represents the high level identity of the entity that was fixed. This is considered to be the source of the defect (e.g. architecture, design, code, requirement)

Count: Number of identified defects, grouped by source

Analysis

Use a bar chart or grouped bar chart to show the number of defects associated with each source. A stacked bar chart can be used to additionally monitor by status. To show trends over time, use a series of bars, or a line chart. Plot by iteration or by phase.

Defect source distribution varies over time, and provides an indication of where development stands in the lifecycle.

Expected trend - Ideally, most architecture defects are identified in very early iterations, tapering off to very few in the last half of the lifecycle with none identified at the end. Requirements defects also peak early in the lifecycle tapering off with a few peaks and valleys to the end. Design defects peak a bit later than architecture but taper off at a slower rate through the second half of the lifecycle. Code and test defects peak later in the process.

Monitor changes in the distribution of defects throughout the lifecycle. Set targets based on development approach and the level of uncertainty in the project. Monitor unexpected peaks to identify issues with specific artifacts or activities, and analyze for potential process improvements.

For example:

  • If a high number of requirements-based defects are found late in the lifecycle, the team should assess their requirements elicitation and traceability processes and tools. If a high number of defects are traced to a specific requirement, there are likely quality issues with that requirement that need to be addressed.
  • When a high number of architecture defects are identified, the team should perform sufficient analysis to prevent costlier defects associated with the architecture from arising late in the lifecycle.

Frequency and reporting

Defect classification should begin in the first iteration, and results reviewed by the team in each iteration assessment. Stakeholders should review the data as each phase milestone is reached. The data is also reviewed at the end of each release as an input to planning for the next release.

Trend reports from projects across the organization can be rolled up to the organization level and tracked over time as process improvements are introduced.

Collection and reporting tools

IBM® Rational® ClearQuest® collects Defect Density by Origin data. It is necessary to implement custom attributes to support the Orthogonal Defect Classification framework. IBM® Rational® Quality Manager® is also used to collect data for this metric.

Assumptions and prerequisites

  • A semantic classification of defects is established and well understood by the team.
  • Upon closure, each defect is assigned a Target value to associate it with its source
  • The team has implemented effective test processes with sufficient coverage to identify defects associated with all sources

Pitfalls, advice, and countermeasures

  • Team members must agree on the source of each defect. As each defect is classified, there must be a clear indication of something missing or incorrect in a specific artifact/ source.
  • Defect classification provides fast feedback to the team, allowing them to address quality issues with very specific artifacts and activities early in the process.
  • It is essential for results to feed back into the process in order to drive improvement. Without this key step, other projects in the organization cannot learn from previous defect patterns and take action to prevent the same mistakes happening in the future. Use the metric to monitor process improvements as they are introduced.
  • A lack of consistency and discipline in classifying defects can lead to skewed results, making trend analysis useless.