Guideline: How to Conduct and Organizational Assessment
This guideline provides advice for conducting a successful assessment of the organization's software delivery capabilities.
Relationships
Related Elements
Main Description

Gather information

The objective is to identify how the organization current approaches software delivery. Information can be gathered through several methods:

  • Analyzing the outcome of previous assessments and projects
  • Interviews with stakeholders and key people
  • Reviews of current software delivery process assets and project artifacts

Conduct a kickoff meeting

Kick off the assessment with stakeholders and key people by giving an overview of the assessment goals and activities. Confirm the scope of the assessment and ask key representatives to give an overview of current software delivery processes. This meeting should not be overly formal or take too much time. A focused 1-2 hour session is usually sufficient.  Make sure to request feedback during and after the session about the approach for conducting the assessment.

Conduct interviews

The objective of conducting interviews is to gather detailed information about how teams across the organization performs software delivery activities, so that you can analyze strengths and weaknesses.

Preparing for the interviews

Before you begin the interviews, confirm the following:

  • Interview schedule
  • Names of those who will conduct each interview and note takers
  • List of interview questions
  • Tools used to record responses, such as:
    • Voice recorders
    • Mind mapping tools e.g. MindManager, Freemind
    • Microsoft® Word® or Excel®
    • IBM® Rational® Software Architect, Rational® RequisitePro®, SoDa
  • How will you structure the results report (e.g. by practice, discipline, priority)
Probe for actual processes

Remember that documented processes may not actually be implemented. Those in use on real projects may be quite different. As you conduct interviews with project teams, confirm that you review real project artifacts and discuss the actual process followed to create them.

During each interview, confirm the following the interviewee:

  • The interviewee's roles and responsibilities.
  • Other roles they interact with to perform their responsibilities.
  • Processes performed.
  • Their opinion on the top 5 problems that currently prevent them to perform their job most effectively.
  • Improvements they think are most worthwhile.
  • Processes they perform.
  • What is currently working well and should not be changed.


Follow the list of specific questions prepared in advance that are based on the particular role of the interviewee. However, be prepared to "go off-script" if the interview leads in a direction that you did not anticipate but is providing useful information.

In interviewing people who are in roles that are above the project level, devote more time within the interview to better understand business drivers and the basic business model. Managers can have differing viewpoints, and executive visions are not always implemented cohesively throughout an organization. You can use the interview questions template to guide you in what questions to ask.

Put interviewees at ease

Some interviewees may see an assessment interview as a threat. They may not be following the prescribed process (many times, for good reason). To be able to capture this type of insight, the interviewees must be comfortable with the interview and be confident that their work habits and views will not have an adverse affect on them.

With this in mind, prepare an interview environment that helps put them at ease:

  • The interview setting should be in a private area, such as a conference room that has been reserved for the interviews.
  • Explain that the purpose of the interview is to assess the current state of affairs and that specific recommendation for improvement will be presented as a result of the assessment. Any interviewees who were not at the kickoff meeting may not have a clue as to why managers asked them to meet with you.
  • Assure the interviewees that no one will be identified as the source of any specific comments.
  • Assure the interviewees that there may be questions that they may not be able to answer, and that is OK. They also may not have data at hand to back up their answers, and that is also OK, but tell them that you will need to get the data later.
  • Explain that you will be taking notes, but that these are for the assessment purposes only and will not be shared outside of the assessment team.
  • Let the interviewees know that they will be given an opportunity at the end of the interviews to ask questions.
  • Explain the type and form of feedback that they should expect to receive as a result of their participation (Findings Presentation, Assessment Report).
Probing for business impact

During interviews, it is easy to see symptoms first rather than the underlying root causes. It is important to uncover the root causes of the problems, inhibitors to success, and the business impact of those problems to be able to prepare credible and compelling findings and recommendations.

Sometimes, you will hear statements that indicate a symptom of an underlying problem, such as "We're always late getting our products out the door" or "Our products don't meet user expectations." You need to probe beneath those statements to find out why these problems occur, at least from the interviewee's perspective. Try to corroborate what you have found by asking other people, in subsequent interviews, who might see the same problem from another perspective.

Other times, you will hear statements that directly express a root cause, such as, "Our problem is that we have no change management process." The lack of an effective change management process is actually a root cause to many problems; therefore, to appropriately prioritize the problem, you need to know what sort of impact it has on the business. It might be that the interviewee felt the delivered product or application didn't meet expectations, or it could be that the project teams are not able to deliver on time. If you can get the person to quantify the impact of the problem, all the better.

Be aware that when you hear someone state that something is a problem, it might not be accurate. They may have a change process, but not be aware of it, not agree with it, or simply not be part of it. Take any suggested solutions seriously, but be sure to verify the information with many interviewees.

You can also probe the interviewees for their opinions on who in the organization is most affected by this problem. That can lead to additional stakeholders who were not on the original list of interviewees but might be able to provide valuable insight.

Closing questions 

When closing the interviews, be sure to give the interviewees a chance to express or summarize their opinions of what changes should and should not be made. Answers to these questions can contribute directly to your key findings in terms of "areas for improvement" and "areas of strength." You can ask, for example:

  • From your vantage point, what changes, if implemented, would have the most significant positive impact on your area?
  • As the organization moves forward, what areas of strength would you like to see preserved?

Review development environment

Project artifacts

Review multiple representative samples of each key artifact if at all possible. It is best to not only determine the state, organization, coherence, and clarity of process definition, but also the state, quality, consistency, and compliance of the related templates and artifacts. This can provide a more objective structure for point-based evaluations according to dimensions derived from these characteristics.

In collecting types of artifacts for review, consider the following categories:

  • Work products
  • Process or methodology definitions and procedural guidelines
  • Planning and management artifacts
  • Policy and governance documents
  • Measurements and metrics
  • Status and reporting artifacts
  • Training materials and user documentation

As you review project artifacts, look at the structure of the artifact and whether it conforms, at least in spirit, to the defined process. Try to get answers to these questions:

  • What is the process for controlling or tracking changes to artifacts? What is the change control board for project artifacts?
  • Are the artifacts are managed in a digital repository or available only on paper?
  • Is each artifact used to add value to their project or just to put a "check in the box" to satisfy the process requirements?
  • Are the artifacts are a by-product of development or the focus of the development? For example, are teams spending a lot of time writing documents rather than building the application?
  • Are the artifacts created as an integral part of the process or as "after the fact" documentation of the process?
  • Can teams automatically trace artifacts from one development phase to another (requirements mapped to test cases and so forth)?
  • Are the processes and artifacts consistent across projects and departments? For example, does one project have a very different requirement set than another project?

Just scan the artifacts. You don't have time for an in-depth review, and that is not the purpose of the assessment. 

Consolidate the interview results

The assessment team analyzes the interview data and supplementary information to produce the initial findings and recommendations. Ambiguities in the findings may necessitate follow-up interviews or access to artifacts. Two key activities are:

  • Analyze and consolidate interview results
  • Create assessment findings and validate in order to gain consensus.

Identify any areas that require further research (e.g. a referenced process website or further key work products that were not originally inspected or available).
Have sufficient domain expertise available in the assessment team to perform the best practice gap analysis when preparing interview findings.