Reduction of Risk Estimation Variance
This metric tracks how much estimation variance is reduced by removing uncertainty in the project.
Relationships
Related Elements
Main Description

Purpose

Reduction of estimation variance is essential for meeting project cost and schedule commitments. At the beginning of a project, cost, effort, and duration estimates are based on information available at that time. The probability of those early estimates exactly matching actual results is not very likely except in the most basic of projects where the problem space and technologies are very simple and well understood.

Estimation variance describes the probability distribution of an estimate (cost, date, or effort), meaning how likely the expected value is to be accurate. This variance is typically greater the further out into the future an estimate is provided. As uncertainty is reduced in the project (e.g. the team learns more about new technologies that are part of the solution and the scope of the effort), estimates become more accurate, and the team has more confidence in them. The earlier in the lifecycle uncertainty is removed, the higher the probability the team will meet or exceed its release commitments.

Estimation variance reduction is a key indicator of successful iterative development adoption. Iterations are planned such that high probability, high impact and architecturally significant risks are addressed early in the lifecycle, driving down uncertainty in the project. As risks are addressed, estimation variance decreases.

Definition

Estimation variance can be computed for any project variable (e.g. effort or cost) in a given iteration or phase.

For each estimated element (e.g. work item, component), the team provides the following values:

  • Best Case Estimate - the most optimistic estimated value (if everything is perfect)
  • Expected Case Estimate - the most likely estimated value
  • Worst Case Estimate - the highest estimated value (if more things go wrong than right)

The amount of uncertainty is reflected in the difference between the Best Case Estimate and the Worst Case Estimate. When the difference is low, there is a high level of confidence in the estimate.

Estimation Variance (for a given iteration or phase) = (sum of all Worst Case Estimates) - (sum of all Best Case estimates) / Number of Estimated Elements

Analysis

A good way to monitor Reduction of Risk Estimation Variance throughout the lifecycle is to use a trend line. Plot Estimation Variance on the Y axis, and iterations or phases on the X axis.

Expected trend - Ideally, estimation variance reduces over the lifecycle for teams focused on risk management. 80% of estimation variance is removed in the early stages of the lifecycle (during inception and elaboration phases) as risks (uncertainty) are removed. During the construction phase 80% of the remaining 20% is removed. As the project progresses the team's confidence in their estimates increases due to a better understanding of the requirements as well as an increasing history of estimation data. The likelihood of on-time delivery is high.

If estimation variance remains stable or increases, the team is not addressing risk. When estimation variance increases during construction, it is likely that the team did not put enough focus on architectural risks. If estimation variance is going down, but not by much, early in the lifecycle the team should focus on architectural and technology risks in order to prevent problems later in the lifecycle.

Frequency and reporting

High level estimates are provided at the beginning of each phase, and refined at the beginning of the iteration (for all work items planned for that iteration). Estimation variance is calculated for each iteration and phase during iteration and phase milestone reviews to help identify trends. When the trend indicates a problem (for example, estimation variance remains high late in the lifecycle) the team should collaborate with stakeholders to discuss outstanding risks and to adjust project plans as necessary.

Collection and reporting tools

Work Item estimates are captured in IBM® Rational® Team Concert®. The assigned owner of the work item enters a Best Case Estimate, Expected Case Estimate, and a Worst Case Estimate. A Schedule Risk Assessment report can be run at the iteration level based on the variance in the estimates. A Reduction of Risk Estimation Variance by Phase report can be generated with a simple spreadsheet tool, or a project planning tool that has attributes for the three levels of estimates.

Pitfalls, advice, and countermeasures

  • Estimates will differ depending on the person who will perform the work. Estimates should be provided by the person who will perform the work in order to be most useful for monitoring estimation variance.
  • Make sure that any tool used to capture estimate data is simple to use and does not introduce too much overhead to the team.
  • Review estimation variance at the end of each iteration. If the team waits until the end of the phase, it may be too late to make needed adjustments.
  • Communicate key constraints and assumptions with estimates.
  • Prioritize risks and address based on likelihood of occurrence and impact.
  • Examples of typical risks teams may encounter include:
    • Schedule
    • Budget
    • Process maturity
    • Requirements churn
    • Staffing issues
    • Architecture complexity
    • COTS and external components
    • Migration complexity, legacy software
    • Technology maturity