“Another 30 defects uncovered yesterday,” reports the QA Lead.  “Ten defects were fixed,” reports the Tech. Lead.  Taken alone these are alarming statistics, and if they persist long enough, the release date would certainly be in jeopardy.

Many software teams struggle through the QA phase with great anxiety as they normally work through the phase without a compass: nothing to tell them whether they are on track or not.  How many defects are left?  Will we be able to fix all the defects before the release date?  Will we find all the defects before the release date?  These are a few of the questions that teams have anxiety about.  It’s only until a few weeks before the planned release date that they come to grips with the realization that they are in trouble.

Measure, Graph and Track Output

Over the years, I’ve developed some tracking techniques that have helped me to control the QA phase of the software development life cycle.  I like to record, track, and graph some key statistics that I use to measure how well the project is progressing to schedule.  The data points are recorded and graphed daily in an Excel spreadsheet and maintained weekly on a historical basis.  Basically, I record the current week’s data daily in the same cell until the end of the week when the last value persists and begin the process in a new cell for the next week.

Based on the behavior of the trends, I make objective decisions for reporting status in the weekly status report.  Red means our fix rate is trending to deliver the project beyond the scheduled release date unless something is done to correct it.    Yellow means that our fix rate is trending off schedule, and if current trends persist, the project will deliver late.  Green, of course, means that the project is delivering on schedule.   I’ll detail the specific criteria for determining status later in this article.

There are 7 variables that I track during the project:

1. Actual defects found as a line plot.
2. Weekly defects found as a bar graph plot.
3. Actual defects left-to-fix as a line plot.
4. Weekly defects fixed as a bar graph plot.
5. Expected defects to find as a straight line.
6. Expected defects left-to-fix as a straight line.
7. Backlog of open defects as a line plot.

To make this all work, during the estimation phase, I estimate the total number of defects to be found and fixed before the product is released.  (I’ll discuss how I estimate these in a future article).  Figure A is an example of a typical graph with all the data points recorded.  The features of the graph as described above are labeled with their corresponding numbers.

**Note, a single click on all graphs in this article enlarges them and then reduces them after they’ve been enlarged.

Figure A. Labeled Defect Tracking Graph

Notice the 2 straight lines that cross each other.  The line starting from zero and increasing over time is the plot of the expected defects to find trend and the line starting from approximately 2000 and decreasing over time until it reaches zero is the expected defects left-to-fix trend.  The reasoning behind the behavior of the lines is that when you start the project all the defects are left-to-fix and when the project completes, there are zero defects left-to-fix. Likewise, when the project starts, none of the defects have been found, and when the project completes, all the defects have been found.

The purpose of these two lines, what I call idealized lines, is to compare and to contrast them with the corresponding actual lines for found and left-to-fix over time.  The behavior of the actual line data plots in comparison to the idealized lines gives you a measure of how well your project is progressing to schedule, and it quantifies the degree that the project is either ahead or behind schedule visually and measurably.

Calculate Project Sigma

When the fix rate is tracking ahead of schedule, the actual defects left-to-fix line plots below the expected defects left-to-fix line, and when the fix rate is tracking behind schedule the actual defects left-to-fix line plots above the expected defects left-to-fix line.  Similarly, when the find rate is tracking ahead of schedule, the actual defects found line plots above the expected defects to find line, and when the find rate is tracking behind schedule, the actual defects found line plots below the expected defects to find line. 

To get a measure of the degree the lines are behind or ahead of schedule, move the end date of the expected lines forward or backward until the actual line plots with the trend nearly identical to the expected line.  Subtract the calculated end date from the original end date. This value is what I term Project Sigma.  It is negative when the project is tracking behind schedule, and it is positive when the project is tracking ahead of schedule.  The larger the absolute value of Project Sigma, the greater your project is ahead or behind schedule. 

Calculate Project Status 

By analyzing the behavior of the actual line plots against the expected line plots it is possible to calculate and report objective and accurate project status.   As the current date approaches the release date, the actual line plots should plot with a narrow band from the expected line plots; similarly, when the project is further away from the release date, the actual line plots can plot with a wide band from the expected line plots and still deliver to schedule.  This behavior reflects the lower defect find and fix productivity early in the project and increased productivity as the project approaches the release date. Given this behavior, the thresholds for reporting status are relaxed further away from the release date and then restricted as  the release approaches the release date.

I recommend to use two thresholds to calculate project status.  The thresholds in use are determined by where the current date falls with respect to the red line date.The project red line date is the date at the point of interesection of the 2 expected lines.   When a project crosses the red line date, it is in the end game where the project team is predominantly testing the application and fixing defects.  Before the  project crosses the red line date, relax the status reporting triggers and restrict the reporting triggers when the date crosses the red line date.  Only the left-to-fix lines are used to calculate project status as the defects left-to-fix  is normally the critical path for the project: you have to find the defects before you can fix them. 

Below are suggested thresholds for calculating status.   Table 1 identifies the triggers to use for reporting status before the red line date, and Table 2 identifies the triggers to use to report status after the red line date.  First, calculate Project Sigma as described earlier.  Assuming the project has crossed the red line date,  find the column where the value of Project Sigma meets the criteria in the header.  When Project Sigma is postive, the status is naturally green, and no table lookup is required.  When Project Sigma is negative, use the absolute value of Project Sigma to index the tables.

What’s important here is the methodology — not the absolute thresholds and values.  Play with this.  You may find that you need to identify three sets of threshold criteria instead of two as I have shown.   Or, the values for indexing into the tables may not be suitable for your corporate culture, so you may decide to tighten them or relax them.  Change anything about this to suit your requirements, but change them using some logical criteria.  Don’t randomly pick a date for which you would begin using a third set of status criteria.  Create an algorithm to identify that date based on the observed behavior of your projects.

Under Estimated Project Signal

One behavior that I’ve repeatedly observed is that when the actual left-to-fix and the actual found lines intersected prior to the expected line’s intersection, it has always indicated early that the initial project estimates are wrong and the project is underestimated. This behavior is natural as most projects are optimistically estimated or, as some would argue, aggressively estimated.  If you see that behavior on your own projects, it would be a good idea to re-estimate the project.  At a minimum re-evaluate the assumptions that you based your schedule upon to confirm if they are still valid.  Chances are things have changed and you haven’t reflected that in your schedule.  This example is illustrated in Figure B.

Figure B. Actual Left-to-Fix and Actual Found Lines Cross Before Expected Lines

Calculating a New Release Date 

If while tracking your project you’ve concluded that you will not deliver on schedule, this method of tracking becomes very valuable for projecting a new release date. To project a new release date when you’ve concluded the project is off the tracks, you first re-estimate the number of defects that you expect to find and fix in the project.  Plug those new values into your Excel spreadsheet. Then extend the expected left-to-fix line until the actual left-to-fix line plots below the expected left-to-fix line.   Provided you haven’t improved your capacity to fix defects, that date is your new release date.  Figure C illustrates a project that is behind schedule, and figure D illustrates the same project with the new projected release date identified using this technique.

Figure C.  Metrics are Trending Behind Schedule

Figure D.  New Delivery Date Calculated

Summary

Using metrics based on measured output is the most reliable method for tracking a project and assessing whether the project is progressing to schedule.  Once my projects enter the test and defect fixing phase, I rely solely on the metrics and techniques described here to report status to the team and management as to how the project is progressing to schedule.   There are essentially two benefits to this approach.  One, it takes the guess work out of assessing your project status with regard to delivering on the release date. With that you can report on your project status objectively and accurately.  Two, it provides for a quantifiable methodology for controlling the productivity of your project team.  Knowing actual find and fix rates, you can add people or request overtime to bring your project in on schedule when you’ve identified a delay.  In a future article, I will describe how to use this information to motivate and focus the team to deliver on the project objectives efficiently and expeditiously.  

I know once you begin using this tool, you will find it invaluable for managing your projects.  I’ve uploaded an example spreadsheet for you to begin using on your own projects (Example Spreadsheet).  It is populated with all the data and graphs used in this article.  Give it a try, and let me know how you make out, and I’ll be happy to answer any questions you have when posted as a comment to this article. I’m sure you will find this technique invaluable as everyone has who I’ve introduced this to in the past.   Good luck!