Emergency Change Management
The procedures you put in place to handle emergency changes should be flexible enough to facilitate rapid resolution of
the problem, including documentation of who is authorized to make emergency changes to the application, and how to get
in touch with these individuals. You should have either a sufficient number of people who can resolve emergencies, or
those people should be easily accessible at all times to prevent a roadblock in the problem resolution process.
It is very important to maintain both communication and the integrity of documentation through an emergency change.
Documenting the steps taken to resolve the problem is of paramount importance.
You should take into consideration whether the change will resolve the existing problem, as well as whether the change
will cause other problems. For an emergency change process, the steps you need to consider are described following.
Issue determination
It is usually obvious when an emergency change is required in a production environment. However, exactly what change is
required may not be immediately clear. When taking corrective action, it is imperative that you implement only one change at a
time. Otherwise, if the problem is resolved by multiple changes, it is impossible to pinpoint which change actually
fixed the problem. Worse, if other problems are introduced, it is impossible to determine which change was the cause
of the new fault. Each change should go through the full process outlined previously before you begin on the next change. If
a change is shown to have no effect, you should back out of it before beginning the next change (the single exception
being when the initial change is a prerequisite to the next change under consideration).
Limited risk assessment
In most cases, the amount of risk assessment done in an emergency situation is directly proportional to the scope of
the change, and inversely proportional to the effect of the outage. Ultimately, risk assessment is the responsibility
of the support person implementing the change. For this, he or she should rely on their own personal experience, as
well as that of associated support personnel. Many of the ideas given in the previous section about change management best
practices can be adapted to the emergency change environment, but on a more limited scale. Finally, as part of the
limited risk assessment, you should determine which users may be affected by the change.
Communication
Although it is not always possible to notify all users of all changes (especially in emergency situations), the user
community will certainly appreciate any warning you can provide. You should also communicate the details of any
emergency changes to the change manager, allowing him to maintain metrics on emergency changes and root causes. The
information may also affect the scheduling or rollout of future changes.
Documentation
Updating documentation is critical to ensure valid, up-to-date information. During unplanned changes, it is easy to
forget to make updates because of the frantic nature of emergencies. However, undocumented change solutions often
result in increased downtime if the solution is unsuccessful.
It helps to document changes before they are made in emergency situations from a central location, perhaps at the
change manager level. If a central support organization does not exist to document changes before they occur, different
individuals may make changes at the same time, not knowing about each other's activities.
Implementation
If the process of assigning risk and documentation occurs prior to the implementation, the actual implementation should
be straightforward. Beware of the potential for change coming from multiple support personnel without their knowing
about each other's changes. This scenario can lead to increased potential downtime and misinterpretation of the
problem.
Test and evaluation
In this phase, the person who initiated the change is responsible for ensuring that the emergency change had the
desired effect and if not, restarting the emergency change process. Steps to take in the investigation of the change
include:
-
Observe and document the impact of the change on the problem.
-
Observe and document any side effects of the change.
-
Determine whether the problem has been resolved, and if so, make sure that all of the necessary documentation occurs to
properly reflect the change.
-
If the change is unsuccessful, back out, and continue the emergency change process until the problem is
resolved or a work-around is in place.
Once the change has been deemed successful, send all emergency change documentation to the change request manager for
review and documentation by the change control team. The change request manager should coordinate a post-mortem
evaluation of the problem to determine potential improvements to prevent future emergency changes of this type. The
change request manager should also bring the change request to engineering or architecture experts for review, and
allow them the opportunity to change solution templates, standard software versions, or designs to better meet the
goals or requirements of your organization.
Performance indicators for Change Management
It is important for you to be able to measure the success of your change management process. The best mechanism is
establishing some performance indicators. These indicators should be reviewed regularly to ensure that change planning
and change management are working well.
Change Management metrics by functional group
Include the percentage and quantity of change success by functional group and risk level in your change management
metrics. Emergency changes should be identified separately in the metrics by functional group, including the success
rate for attempted fixes. You should contact the user identified on the change request form following the change to get
an understanding of change success.
Change history archive
The change coordinator is also responsible for archiving the change history. For archival purposes, creating a spreadsheet with columns for
functional group success and failure, and rows for the months, is sufficient. Change history archives can
help identify current issues based on past change rates and available resources. The information can also be used to
investigate change rates in general for overall planning purposes.
Targeting change success
To target change success, you should start with a baseline of change management metrics. The change coordinator can then
identify potential issues and set overall goals.
Periodic Change Management performance meeting
It is important to review the metrics you collect monthly, including the following metrics:
- Change quantity and risk level
- Change failure quantity and post mortem evaluations
- Emergency changes and post mortem evaluations
- Change management goals
- Undocumented changes
.
The change request manager should review the metrics and report to the
appropriate teams for improvement.
|