You cannot control what you cannot measure.
- Tom DeMarco (Software Engineer)

How does the software community improve the control over managing our software development projects?  Ken Schwaber in “Agile Software Development with SCRUM” suggests that the only way to control software development with so much complexity and unpredictability is with an empirical process control model.

Tunde told me the empirical model of process control, on the other hand, expects the unexpected.  It provides and exercises control through frequent inspection and adaptation for processes that are imperfectly defined and generate unpredictable and unrepeatable outputs.  He recommended I study this model and consider its application to the process of building systems.

– Ken Schwaber, Agile Software Development with SCRUM (Series in Agile Software Development)

I happen to agree with Ken’s position on the need for an empirical approach to software process control.  While so much of what is offered by the Agile community is presented as new, the basis of CMM(I) is empirical.  Not only does CMM(I) require empirical control of the software project itself, it requires empirical analysis of the process as well.  Metrics are the basis for all process improvement and project control in the CMM(I) paradigm.

Background

It was during my first experience with CMM(I) process improvement in 1998 when the value of software metrics became apparent to me.  It happened quite accidentally.  I was neutral on the process that we developed based on CMM(I), but I was determined to be a good soldier and give it my support; after all, the company had invested a great deal of money in this initiative.  The least that I could do is to try my best to make it work.  And that’s when the light turned on for me to empirical process control.

CMM(I) identifies two requirements that helped me to arrive at this insight: the basis of estimating is to first estimate the size and then to transform size into duration with a productivity rate; the second requirement is to track completed size as well as completed time. 

Before this, I did not have the tools to accurately and reliably understand how well the project was progressing to schedule.

Why Measure Size?

When you contrast completed size against completed time, problems light up like a Christmas tree.   For example, if completed size on a project is 25% and completed time is 75%, the schedule problem is obvious.  When project management is done well, it’s all about frequent inspection and adaptation, and as Tunde explained to Schwaber, this is the basis for the exercise of control over software projects.  

Tracking with metrics identifies incorrect estimating assumptions early, and consequently, it gives you, the project manager, the control to deliver on the release with no changes in commitments even when the project estimates are incorrect.  That’s the power of an empirical process control model. 

My reservation with Scrum is that the fidelity of the empirical model is low.  You’ve probably experienced it on your software projects. One of the developers on an assignment informs the team that they are done with one of the deliverables.  Two weeks later, the developer continues working on the assignment.  The right metrics will confirm when done is done.

Scrum’s low fidelity is a consequence of only measuring effort remaining.  Measuring both effort and size is analogous to measuring distance to a distant object using the parallax method of measurement.  Measuring the base angles of the triangle is only an interesting observation, but when you add the distance between the base angles, you have the missing information to precisely measure the distance to that distant object.  Similarly, when you contrast completed effort against completed size, you have the necessary information to make accurate inferences about your project’s status with respect to schedule.

In this essay, I’ll explain one aspect of an empirical model that demonstrates the 6-point process described in my previous essay “Software Metrics: An Example Approach.”

Project Conditions 

• You’ve estimated 600 defects to find and fix for a six-month project delivery. 
• The project is 8 weeks away from delivery with 600 defects having been found and with only 200 defects left to fix. 400 defects have already been fixed.
• To reserve time for the unexpected, the manager sets a target to complete fixing the 200 remaining defects in 6 weeks.
• There are 10 developers on the project assigned to defect fixing.
• Project history calibrates average defect fix rate at 1 defect per 1.5 staff days.  Just in case this concerns some readers, this average includes defects that take 10 or more staff days to fix as well as ones that take only hours to fix.  If there are enough defects, we only need to concern ourselves with the mean as the Law of Large numbers tells us that the team will perform at the expected value, which is the mean.
• To keep the example simple for illustrative purposes, the total required defects to fix remains constant during the 6 week period, and developers do not introduce new defects with their fixes, and they fix defects correctly the first time.  While this never happens in practice, introducing these variables into the example unnecessarily complicates the presentation.

Apply the Six Step Process

1. Identify what done looks like with an objective measure.
   1. Inform the team that they have six weeks to fix 200 defects.
2. Break the big goal into smaller goals. 
   1. To achieve the release goal, the team needs to fix 34 defects a week. 
3. Give the team physical productivity goals. 
   1. I like to give stretch goals, so I’ll ask the team to fix 40 defects for the week.
   2. It’s also important to give individual goals because team success is based on individual success.  To fix 40 defects in a week, each developer needs to work towards fixing 4 defects for the week.
   3. Here’s an example of what I’ll say to the team:  “I’d like each developer to target fixing 4 or more defects a week. I realize some of you may have a difficult defect and fix only 1 or even none, and others will have very easy defects and fix 10 or more.  We aren’t being measured on our productivity as individuals, but on our productivity as a team.  If you exceed your individual target but we fail our team target, we failed as a team and individuals.  If we all shoot to exceed our target, the team will easily have achieved the required target of fixing 34 defects a week to deliver on time.”
4. Measure physical progress towards done.
   1. At the end of the week, count how many defects were fixed.  For this example, the team was successful in fixing 38 defects in the past week.
5. Communicate to the team the physical progress towards done and what production is remaining to achieve done.
   1. Last week we were successful in fixing 38 defects.  We are ahead of our goal.  We have 5 weeks left to fix 162 defects. 
6. Repeat starting at step 2 until the overall objective has been achieved.

As is often the case in a real project, additional defects are discovered during this 6 week period, and defects are reopened as they are not fixed correctly.   As a consequence, analysis of the metrics will likely indicate that the original goal cannot be met. In this case at step five, the team needs to make some choices: 

• Defer defects to a future release.
• Add more engineers to the project for defect fixing.
• Extend the release date.

Once the team has decided on a course of action, a new goal is established, and the process repeats beginning at step 1. 

Benefits of the Process

• Everyone on the team knows exactly what is expected of them to achieve success.  Consequently, team members focus more acutely on their deliverables for the week and guard themselves from needless distractions during the day.  People are more productive when they know what’s expected of them.  Having measurable targets enhances communication of expectations.
• Team members are more satisfied in their work.  Having clear and specific goals is motivating to individuals.  Having clear and objective project status is satisfying to the team and to management.
• There’s more collaboration since the only goal that matters is the team goal.  It’s in the interest of team members to help each other to achieve the team goal.
• Status reporting is more accurate, and you can give status with more confidence.  It’s blatantly obvious when productivity is not tracking to schedule.  For example, if only 60 defects are fixed in total after the first two weeks, it’s obvious that team productivity will not complete the work on time unless something changes.
• Once a schedule problem is determined, analysis of the metrics gives you more certainty of what’s required to put the project back on schedule.  Following from the previous bullet, mean productivity for fixing a defect is established at 1.67 staff days.   With that input, the team can accurately decide on extending the target date, deferring defects, adding additional staff, or any combination of the former.

It’s valuable to note when a data trend emerges, it’s rare for it to change – at least not significantly – so don’t expect the mean effort to fix a defect to improve significantly.  Expecting things to change is a sure way to miss the target because when you discover that nothing changed, you will have less time to apply corrective actions.

Nothing Goes to Plan

It is my experience for software projects to always estimate optimistically.  There always seems to be more work than planned, there always seems to be more defects than planned, and there always seems to be more time required to complete assignments than planned.

The strength of an empirical model of process control is when things don’t go as planned.  When more defects are found than planned, analyzing the metrics tells management there is a problem early, the extent of the problem, and what precise options are available to put the project back on schedule.

Summary

To apply the 6-step process successfully, the following data requirements must be met. 

• The size of a deliverable must be estimated and measured.  The more granular the unit of size, the more control you will have over your schedule.  While I have my preference, any unit of size that is measurable works.
• Effort must be calculated from size by dividing it by a productivity rate.

Software metrics are often a controversial subject in the software industry.  I suspect because of the fear metrics will be misused to measure individuals: opposition to software metrics often revolves around the problem of individual productivity.  Software metrics, when used properly, is a tool for measuring the project not individuals; it’s a tool for controlling project deliverables, assessing and communicating status, and making better decisions.