Guideline: Assessing the Development Organization
This guideline provides specific techniques and advice for performing a development organization assessment.
Relationships
Main Description

Introduction

Development organization assessments are performed in order to understand an organization's current software development capabilities.  The assessment team learns how the organization currently approaches software development and delivery and identifies current process weaknesses and strengths. The assessment team then makes improvement recommendations that are aligned with the organization's operational objectives and validates them with stakeholders.

Factors to consider

When performing an assessment of the organization's current software capabilities, consider the following factors which should shape the character of the organization's processes and tools usage, and their required tailoring guidelines.

Operational Objectives

A successful assessment requires understanding the organization's operational objectives. These objectives are internally-focused, concentrating on operations within the company and their ability to perform the most critical functions to meet business obligations. Operational objectives typically target these criteria:

  • Productivity or efficiency
  • Time to delivery
  • Quality
  • Predictability
  • Value and customer satisfaction

Organizational processes should be defined to align with the organization's operational objectives. For example, if quality is a priority, confirm that the organization has sufficient practice and tool support to prevent and detect defects.

Prior to beginning the assessment, review and validate the operational objectives. 


Business context

There are different types of business contexts that affect the required level of ceremony, the level of formality, and the rigidity of the process. The more stakeholders (e.g. buyers, customers, subcontractors, regulatory bodies) involved, the more likely projects will need to produce formal evidence, such as documents, reports, and prototypes, at major project milestones. Examples to consider:

  • Contract work where the developer produces software to a given customer specification and for this customer only.
  • Speculative or commercial development where the developer produces and covers the cost of putting the software on the market.
  • Internal projects where customer and developer are in the same organization.


Size of typical software development efforts

The typical effort's size will also affect the level of ceremony, the level of formality, and the rigidity of the process. The larger the project, the larger the development team and, regardless of the business context, the more formality and visibility the various teams and management need to have in requirements, interfaces, and progress indicators. Communication issues on large projects are further aggravated by geographically dispersed teams.

The size of software development efforts can be defined by certain metrics such as Source Lines of Code (SLOC), Delivered Source Instructions or Functions Points, number of person-months or merely the cost.

Degree of novelty

The degree of novelty is based on what has preceded a given software effort within the development organization and, in particular, whether the development is in a second or subsequent cycle. This includes the maturity of the organization and its process, its assets, its current skill set, and issues such as assembling and training a team, acquiring tools, and other resources.

A given project's degree of novelty affects the process in a completely different way. For a new type of project, the first of its kind, the inception and elaboration phases will be longer, and may span several iterations. Also, more emphasis will be put on eliciting and capturing requirements, on use-case modeling, on architecture, and on mitigating risk. For a project that is an evolution cycle from a previous system, change management is more crucial and incorporating legacy code poses some technical challenges.

Novelty is not only relative to the system being developed, it's also relative to the maturity of the performing organization because introducing new techniques or tools affects the process.


Type of applications

There are different types of applications (e.g.  embedded real-time systems, distributed information systems, telecom systems, Computer-Aided Software Engineering (CASE) tools). The type of applications delivered by an organization will affect its processes, especially with respect to specific constraints the domain may impose on the development such as safety, performance, internationalization, memory constraints, and so forth.

The type of application may affect the process if the application is mission-critical; for example, the flight-control system in an airplane. A mission-critical system requires a higher level of ceremony in general, both to trace requirements and to assure the quality of the product. A mission-critical application also requires that more resources are spent on testing.

The type of development, or the target domain, bring in process issues such as:

  • Techniques and tools to support specific activities; for example, automatic code generation for finite-state machines.
  • Certification procedures; for example, for medical instrumentation.
  • Compliance to standards; for example, for accounting or fiscal issues, and for telecommunication equipment.


Type of development

There are various types of development, such as:

  • Contract work where you develop a product for a specific customer. You have more stakeholders to manage and negotiate with hen you perform contract work. There is often a need for more formal external artifacts because the customer, or representatives, want to monitor progress and be kept informed. Make sure that the artifacts the customer reviews are easy to understand. Sometimes, there is a need to have a milestone where the project can offer a fixed-price on the rest of the project. In that case, you may need to add a new milestone or adjust the existing milestones. In other cases, you may have to adjust to the lifecycle model the customer is using with other milestones and phases.
  • Speculative development where you develop a product for a mass-market. In speculative development, a marketing (product) manager, within the organization itself, acts as the customer. Time-to-delivery is often more important than the functionality in speculative development and there is less need for formal reviews.
  • Internal development where you develop a product that is delivered to another department within the company. It may be acceptable to be less formal when describing artifacts because they will be reviewed by peers.
  • Pre-studies where you do not normally develop a product. The purpose of a pre-study project is to find out whether it is possible to build a product. A pre-study project doesn't have the same milestones as a typical project.

Technical and managerial complexity

Different types of systems, and their projects, can be classified in terms of the technical complexity of the system and the managerial complexity. For example, a typical small business spreadsheet application is often of low technical complexity and is easy to manage. The other extreme is a typical weapon system project, which is often both technically complex, and complex to manage.

Usually increasing system size, project duration or business context also increases the managerial complexity. Increasing the novelty, in either the problem domain or the solution space, increases the technical complexity. There is an interaction between managerial and technical complexity as well, as many large projects are also technically complex. This results in:

  • Increased managerial complexity that leads to more ceremony, including more formal reviews and milestones, and more artifacts.
  • Increased technical complexity that leads to the introduction of specific techniques, roles and tools, and, therefore, more activities.

Problems and root causes

An important aspect of understanding the software development organization is to understand the problems in the existing software development process. This influences those areas of the process you will concentrate on in the beginning of the process implementation. It is important to note that, if there is no established way of working in the organization, it may be pointless to find problems. Instead, you may need to identify the root causes of the problems. To eliminate the problems, you will tackle the root causes by improving the process, introducing tools to automate the process, and training people.

Examples of common problems:

  • Inability to manage scope-the organization routinely tries to do more than they actually do in the end.
  • Inability to capture requirements-they have difficulty specifying requirements.
  • Inability to manage changing requirements.
  • Inability to manage requirements-requirements do not make it to the final product.
  • Inability to estimate-they are routinely too optimistic about their ability to deliver on schedule.
  • Design deficiency-they are good at meeting requirements, yet poor at designing systems. What kinds of design problems do they have? Are the systems difficult to maintain and enhance? Do they have performance problems, usability problems, capacity problems, and so on?
  • Inability to produce quality products-the product has too many defects which may be due to lack of testing, but usually is also related to an inability to capture and manage requirements, as well as design deficiency.
  • Unacceptable software performance.
  • Low usability.
  • Colliding developers-there is a lack of control over ownership and configuration management, so that developers make conflicting changes and work is lost.
  • Late discovery of problems.
  • Trouble going from use cases to design.

Examples of root causes

A problem is often a symptom that something is wrong. You need to identify the root causes of the problems. The following are examples of some common root causes:

  • Insufficient requirements management
  • Ambiguous and imprecise communications
  • Brittle architectures
  • Overwhelming complexity
  • Undetected inconsistencies among requirements, designs, and implementations
  • Insufficient testing
  • Subjective project status assessment
  • Delayed risk reduction due to waterfall development
  • Uncontrolled change propagation
  • Insufficient automation
  • No systematic way to build user interfaces
  • No way to go from use cases to a design

Advice for conducting the assessment

  • Depending on the scope of the assessment, it can be useful to schedule a kickoff meeting to review and discuss the assessment and the process. Invite stakeholders, participants, and the assessment team.
  • Confirm there is sufficient domain expertise available on the assessment team to identify strengths and weaknesses. Confirm the scope of the assessment
  • Confirm that you have identified the right people to provide the materials that will be assessed. Be clear about what is being requested. Success depends on the availability of information and open communication about what is working and what is not.
  • Prepare interview questions in advance so that the interviews are conducted as consistently as possible.
  • Workshops are often used to support the assessment of an organization's software development processes and tools. See Guideline: Conduct an Assessment Workshop.
  • Perform information gathering on-site through a series of interviews, artifact reviews, and inspections of the development environment