Purpose
Reduction of estimation variance is essential for meeting project cost and schedule commitments. At the beginning of a
project, cost, effort, and duration estimates are based on information available at that time. The probability of those
early estimates exactly matching actual results is not very likely except in the most basic of projects where the
problem space and technologies are very simple and well understood.
Estimation variance describes the probability distribution of an estimate (cost, date, or effort), meaning how likely
the expected value is to be accurate. This variance is typically greater the further out into the future an estimate is
provided. As uncertainty is reduced in the project (e.g. the team learns more about new technologies that are part of
the solution and the scope of the effort), estimates become more accurate, and the team has more confidence in them.
The earlier in the lifecycle uncertainty is removed, the higher the probability the team will meet or exceed its
release commitments.
Estimation variance reduction is a key indicator of successful iterative development adoption. Iterations are planned
such that high probability, high impact and architecturally significant risks are addressed early in the lifecycle,
driving down uncertainty in the project. As risks are addressed, estimation variance decreases.
Definition
Estimation variance can be computed for any project variable (e.g. effort or cost) in a given iteration or phase.
For each estimated element (e.g. work item, component), the team provides the following values:
-
Best Case Estimate - the most optimistic estimated value (if everything is perfect)
-
Expected Case Estimate - the most likely estimated value
-
Worst Case Estimate - the highest estimated value (if more things go wrong than right)
The amount of uncertainty is reflected in the difference between the Best Case Estimate and the Worst Case Estimate.
When the difference is low, there is a high level of confidence in the estimate.
Estimation Variance (for a given iteration or phase) = (sum of all Worst Case Estimates) - (sum of all Best Case
estimates) / Number of Estimated Elements
Analysis
A good way to monitor Reduction of Risk Estimation Variance throughout the lifecycle is to use a trend line. Plot
Estimation Variance on the Y axis, and iterations or phases on the X axis.
Expected trend - Ideally, estimation variance reduces over the lifecycle for teams focused on risk
management. 80% of estimation variance is removed in the early stages of the lifecycle (during inception and
elaboration phases) as risks (uncertainty) are removed. During the construction phase 80% of the remaining 20% is
removed. As the project progresses the team's confidence in their estimates increases due to a better
understanding of the requirements as well as an increasing history of estimation data. The likelihood of on-time
delivery is high.
If estimation variance remains stable or increases, the team is not addressing risk. When estimation variance increases
during construction, it is likely that the team did not put enough focus on architectural risks. If estimation variance
is going down, but not by much, early in the lifecycle the team should focus on architectural and technology risks in
order to prevent problems later in the lifecycle.
Frequency and reporting
High level estimates are provided at the beginning of each phase, and refined at the beginning of the iteration (for
all work items planned for that iteration). Estimation variance is calculated for each iteration and phase during
iteration and phase milestone reviews to help identify trends. When the trend indicates a problem (for example,
estimation variance remains high late in the lifecycle) the team should collaborate with stakeholders to discuss
outstanding risks and to adjust project plans as necessary.
Collection and reporting tools
Work Item estimates are captured in IBM® Rational® Team Concert®. The assigned owner of the work item enters a Best
Case Estimate, Expected Case Estimate, and a Worst Case Estimate. A Schedule Risk Assessment report can be run at the
iteration level based on the variance in the estimates. A Reduction of Risk Estimation Variance by Phase report can be
generated with a simple spreadsheet tool, or a project planning tool that has attributes for the three levels of
estimates.
Pitfalls, advice, and countermeasures
-
Estimates will differ depending on the person who will perform the work. Estimates should be provided by the person
who will perform the work in order to be most useful for monitoring estimation variance.
-
Make sure that any tool used to capture estimate data is simple to use and does not introduce too much overhead to
the team.
-
Review estimation variance at the end of each iteration. If the team waits until the end of the phase, it may be
too late to make needed adjustments.
-
Communicate key constraints and assumptions with estimates.
-
Prioritize risks and address based on likelihood of occurrence and impact.
-
Examples of typical risks teams may encounter include:
-
-
Schedule
-
Budget
-
Process maturity
-
Requirements churn
-
Staffing issues
-
Architecture complexity
-
COTS and external components
-
Migration complexity, legacy software
-
Technology maturity
|