Purpose
This metric indicates the quality of the system, based on the number of defects found and according to severity of the
defects. Definition
Definition
Count - Number of defects by severity.
Group by:
-
Severity - defect severity ratings are based on the organization's standard (e.g. high, medium, and low
severity)
-
Pre-shipment and post-shipment
-
Component
Analysis
A good way to monitor Defect Distribution is to use a line chart. Plot number of defects on the Y axis and time/
iterations on the X axis.
Group defects by component to determine whether any particular component has quality issues. Group by pre-shipment
and post-shipment defects to determine the quality of the system before and after the product has been shipped.
-
Expected Trend - generally, the trend line should slope downward later in the project
lifecycle
-
Low numbers of defects - a low value may indicate good quality or insufficient testing.
-
Upward slope - this trend may indicate a problem area to investigate, or could be a
good result due to increased testing.
For Agile and/ or iterative development approaches, testing starts early in the development lifecycle.
-
An upward trend of defects found (cumulative) from the beginning of the lifecycle is expected
-
The number of defects found (in each iteration) should be consistent across the lifecycle. Expect a reduction
in defects found at the end of lifecycle (if testing has been performed properly)
-
The number of closed defects should be equal or very close to the number of defects found in that
iteration (i.e. the opened defects should burn down to zero)
For waterfall or non-iterative development projects, the test cycle usually begins late in the development lifecycle.
-
The number of defects found trends upward toward the middle to the end of the development lifecycle. The
defect discovery rate slows down at the end of development. Expect a bell curve for defect arrival.
-
The number of opened defects trends downward to zero when development is coming to an end.
Closely monitor high severity defects which can make the product unstable. Focus on removing these high severity
defects first. The product should not be delivered to the customer if there are still high severity defects in the
system.
The following example report groups defects by component (Online auction project under Smarter Planet program).
The following graph shows an example of a defect distribution be severity report.
Collection and reporting tools
IBM® Rational® Project Conductor® collects Defect by Severity data at the project and program level. IBM®
Rational® ClearQuest Enterprise®, IBM® Rational® Team Concert®, and IBM® Rational® Quality Manager® also
collect Defect Distribution data. IBM® Rational® Insight® reports this metric.
Pitfalls, advice, and countermeasures
-
Use the Test Trend (Functional and System) metric as a countermeasure for Defect Distribution to ensure that
sufficient testing has been performed.
|