Task: Assess the Measurement System
This task is for evaluating whether the infrastructure and the process are meeting the needs of the business and IT organization.
Disciplines: Project Management
Purpose
To evaluate the measurement system in light of previously defined organizational and business goals.
Relationships
Main Description

The impacts of changing individual and team behavior to achieve business and organizational objectives probably will not be apparent for a much longer time than it takes for the effects on individuals and teams to become apparent. Because of this, it is important that everyone implementing and using the measurement system understand this. The success of an implementation of a measurement system should never be determined in anything less than eight months to a year. It takes time for changes at the lowest level of an organization to show sustained benefit at the organization and business levels.

The results and recommendations are usually documented in a Measurement Assessment.

Steps
Assess the Tools Configuration and Infrastructure

The assessment of the infrastructure for the measurement system is the easiest element to evaluate. The tools and their configuration either work or they do not. They either efficiently support automatic collections or they do not. Take corrective action immediately to resolve any issues associated with infrastructure. Languishing infrastructure issues (connection, performance, down time, and so forth) kill any attempt at establishing a measurement system.

Organizations have much fewer problems with the infrastructure if they deploy the measurement system to the organization incrementally and pilot the use of each increment before moving the contents out to the broader population. Some metrics will not be affected. However, those showing trends over time are extremely vulnerable to lack of confidence when there are issues that arise that could have been resolved on a small scale early.

The longer it takes to show growth toward organizational and business objectives, the less likely the measurement system will continue.

Assessing the configuration and infrastructure should include items such as:

  • Issues with the hardware and software for the measurement system 
  • Issues with hardware down time
  • Issues with availability of metric reports
  • Correctness of the derived measures 
  • Usability of the metric reports
  • Timeliness of the metric reports
Assess the Measurement Process

The measurement process (who, what, how, and when) needs to be evaluated regularly. As businesses and IT organizations change, so should the content of their multi-tiered measurement system that supports them. As the business changes, or as IT changes, the focus may shift from one set of organizational objectives to another. To measure the impact of the changes associated with these new objectives, an additional or a different set of measures may need to be used.

A note of caution:
Changing a measurement system too much, too soon or too often will send the message that the newest measurement set is simply another measure of the month. Before deploying too much change too soon, it is advisable that the current set of measures have demonstrated achievement of the goals set for them (for example, institutionalizing a set of practices and techniques, deployment of a suite of tools across the entire organization, and so on).

Assess Team Usage Models

Assess the understanding of those who are using the deployed metrics. This includes understanding and proper use at all levels of the IT organization. The effectiveness of a deployed performance system increases with its use; nonetheless, many attempts at sustaining a measurement system have failed due to the lack of use and commitment.

Any metric not used should be removed. Therefore, project teams and management must use the metrics for what they were intended. The purpose of a performance measurement system is to provide a mechanism for reasoning about the effectiveness of a change effort and the efficiency of tools and people towards achievement of business goals. Periodic assessments should occur if for no other reason than it sends a clear statement from management that the measurement system is important. This will not allow it to become forgotten and useless.

Not having a vibrant multi-tiered measurement system gives rise to the questions of value that are directed internally toward organizations as a whole and at people, processes, and tools in particular. To create and maintain a vibrant measurement system requires commitment by everyone at all levels in IT. A programmer or a technical writer may not be that interested in a metric about meeting milestones or a metric related to project financials, but they should be made aware of what changes they may need to make or that the organization needs to make to create a flexible organization that provides value to the business. It is the business that provides the work environment, and it is the role of IT to support the business in meeting its objectives.