Defect Trends
This metric tracks the rate of defect arrival and closure (fixing) over the project duration.
Relationships
Related Elements
Main Description

Purpose

Monitor Defect Trends to ensure that arrival and closure rates have some correlation (i.e. that your arrivals do not consistently outpace your closure, resulting in a high defect backlog). You can also use this metric to determine the remaining defect backlog, and to project the future defect arrival and closure rate up to and including the post-ship phase.

Definition

Count

  • Number of defects found and closed each unit of time (usually a week, but could be a day or month, depending on your iteration length). This is a graph over time, typically in two lines.

You can also superimpose bars of remaining defects over the line graphs, because this may not always be evident. (The close rate can exceed the arrival rate for one or more data points, which can obscure the backlog.)

Analysis

As the end of cycle (ship date) nears, you should expect to see the arrival rate slowing and the closure rate surpassing it. This indicates that fewer defects are being found (hopefully because you have found them already), and that any backlog or outstanding defects are being fixed. If the arrival curve does not taper down, it suggests that you may continue to find defects post-ship. In the cumulative chart, you expect to see the gap between total found and total fixed. If a large gap remains, you have many defects that are still not closed: this may necessitate lengthening the schedule or shipping with unclosed defects.

Note that looking purely at arrivals and closures does not include defects in a backlog that predates the data collection.

Considerations based on development approach

For Agile and/ or iterative development approaches, testing begins early in the development lifecycle.

  • An upward trend of defects found (cumulative) from the beginning of the lifecycle is expected
  • The number of defects found (in each iteration) should be consistent across the lifecycle. Expect a reduction in defects found at the end of lifecycle (if testing has been performed properly)
  • The number of closed defects should be equal or very close to the number of defects found in that iteration (i.e. the opened defects should burn down to zero)

For waterfall or non-iterative development projects, the test cycle usually begins late in the development lifecycle.

  • The number of defects found trends upward toward the middle to the end of the development lifecycle. The defect discovery rate slows down at the end of development. Expect a bell curve for defect arrival.
  • The number of opened defects trends downward to zero when development is coming to an end.
     

The following graph is an example of a defect trends report.

defect trends

Use this metric in conjunction with Defect Density.

Collection and reporting tools

IBM® Rational® Quality Manager® collects defect arrival and resolution (by Test Plan). IBM® Rational® Insight® reports on this metric.