You cannot control what you cannot measure.
- Tom DeMarco (Software Engineer)

How does the software community improve the control over managing our software development projects?  Ken Schwaber in “Agile Software Development with SCRUM” suggests that the only way to control software development with so much complexity and unpredictability is with an empirical process control model.

Tunde told me the empirical model of process control, on the other hand, expects the unexpected.  It provides and exercises control through frequent inspection and adaptation for processes that are imperfectly defined and generate unpredictable and unrepeatable outputs.  He recommended I study this model and consider its application to the process of building systems.

– Ken Schwaber, Agile Software Development with SCRUM (Series in Agile Software Development)

I happen to agree with Ken’s position on the need for an empirical approach to software process control.  While so much of what is offered by the Agile community is presented as new, the basis of CMM(I) is empirical.  Not only does CMM(I) require empirical control of the software project itself, it requires empirical analysis of the process as well.  Metrics are the basis for all process improvement and project control in the CMM(I) paradigm.

Background

It was during my first experience with CMM(I) process improvement in 1998 when the value of software metrics became apparent to me.  It happened quite accidentally.  I was neutral on the process that we developed based on CMM(I), but I was determined to be a good soldier and give it my support; after all, the company had invested a great deal of money in this initiative.  The least that I could do is to try my best to make it work.  And that’s when the light turned on for me to empirical process control.

CMM(I) identifies two requirements that helped me to arrive at this insight: the basis of estimating is to first estimate the size and then to transform size into duration with a productivity rate; the second requirement is to track completed size as well as completed time. 

Before this, I did not have the tools to accurately and reliably understand how well the project was progressing to schedule.

Why Measure Size?

When you contrast completed size against completed time, problems light up like a Christmas tree.   For example, if completed size on a project is 25% and completed time is 75%, the schedule problem is obvious.  When project management is done well, it’s all about frequent inspection and adaptation, and as Tunde explained to Schwaber, this is the basis for the exercise of control over software projects.  

Tracking with metrics identifies incorrect estimating assumptions early, and consequently, it gives you, the project manager, the control to deliver on the release with no changes in commitments even when the project estimates are incorrect.  That’s the power of an empirical process control model. 

My reservation with Scrum is that the fidelity of the empirical model is low.  You’ve probably experienced it on your software projects. One of the developers on an assignment informs the team that they are done with one of the deliverables.  Two weeks later, the developer continues working on the assignment.  The right metrics will confirm when done is done.

Scrum’s low fidelity is a consequence of only measuring effort remaining.  Measuring both effort and size is analogous to measuring distance to a distant object using the parallax method of measurement.  Measuring the base angles of the triangle is only an interesting observation, but when you add the distance between the base angles, you have the missing information to precisely measure the distance to that distant object.  Similarly, when you contrast completed effort against completed size, you have the necessary information to make accurate inferences about your project’s status with respect to schedule.

In this essay, I’ll explain one aspect of an empirical model that demonstrates the 6-point process described in my previous essay “Software Metrics: An Example Approach.”

Project Conditions 

• You’ve estimated 600 defects to find and fix for a six-month project delivery. 
• The project is 8 weeks away from delivery with 600 defects having been found and with only 200 defects left to fix. 400 defects have already been fixed.
• To reserve time for the unexpected, the manager sets a target to complete fixing the 200 remaining defects in 6 weeks.
• There are 10 developers on the project assigned to defect fixing.
• Project history calibrates average defect fix rate at 1 defect per 1.5 staff days.  Just in case this concerns some readers, this average includes defects that take 10 or more staff days to fix as well as ones that take only hours to fix.  If there are enough defects, we only need to concern ourselves with the mean as the Law of Large numbers tells us that the team will perform at the expected value, which is the mean.
• To keep the example simple for illustrative purposes, the total required defects to fix remains constant during the 6 week period, and developers do not introduce new defects with their fixes, and they fix defects correctly the first time.  While this never happens in practice, introducing these variables into the example unnecessarily complicates the presentation.

Apply the Six Step Process

1. Identify what done looks like with an objective measure.
   1. Inform the team that they have six weeks to fix 200 defects.
2. Break the big goal into smaller goals. 
   1. To achieve the release goal, the team needs to fix 34 defects a week. 
3. Give the team physical productivity goals. 
   1. I like to give stretch goals, so I’ll ask the team to fix 40 defects for the week.
   2. It’s also important to give individual goals because team success is based on individual success.  To fix 40 defects in a week, each developer needs to work towards fixing 4 defects for the week.
   3. Here’s an example of what I’ll say to the team:  “I’d like each developer to target fixing 4 or more defects a week. I realize some of you may have a difficult defect and fix only 1 or even none, and others will have very easy defects and fix 10 or more.  We aren’t being measured on our productivity as individuals, but on our productivity as a team.  If you exceed your individual target but we fail our team target, we failed as a team and individuals.  If we all shoot to exceed our target, the team will easily have achieved the required target of fixing 34 defects a week to deliver on time.”
4. Measure physical progress towards done.
   1. At the end of the week, count how many defects were fixed.  For this example, the team was successful in fixing 38 defects in the past week.
5. Communicate to the team the physical progress towards done and what production is remaining to achieve done.
   1. Last week we were successful in fixing 38 defects.  We are ahead of our goal.  We have 5 weeks left to fix 162 defects. 
6. Repeat starting at step 2 until the overall objective has been achieved.

As is often the case in a real project, additional defects are discovered during this 6 week period, and defects are reopened as they are not fixed correctly.   As a consequence, analysis of the metrics will likely indicate that the original goal cannot be met. In this case at step five, the team needs to make some choices: 

• Defer defects to a future release.
• Add more engineers to the project for defect fixing.
• Extend the release date.

Once the team has decided on a course of action, a new goal is established, and the process repeats beginning at step 1. 

Benefits of the Process

• Everyone on the team knows exactly what is expected of them to achieve success.  Consequently, team members focus more acutely on their deliverables for the week and guard themselves from needless distractions during the day.  People are more productive when they know what’s expected of them.  Having measurable targets enhances communication of expectations.
• Team members are more satisfied in their work.  Having clear and specific goals is motivating to individuals.  Having clear and objective project status is satisfying to the team and to management.
• There’s more collaboration since the only goal that matters is the team goal.  It’s in the interest of team members to help each other to achieve the team goal.
• Status reporting is more accurate, and you can give status with more confidence.  It’s blatantly obvious when productivity is not tracking to schedule.  For example, if only 60 defects are fixed in total after the first two weeks, it’s obvious that team productivity will not complete the work on time unless something changes.
• Once a schedule problem is determined, analysis of the metrics gives you more certainty of what’s required to put the project back on schedule.  Following from the previous bullet, mean productivity for fixing a defect is established at 1.67 staff days.   With that input, the team can accurately decide on extending the target date, deferring defects, adding additional staff, or any combination of the former.

It’s valuable to note when a data trend emerges, it’s rare for it to change – at least not significantly – so don’t expect the mean effort to fix a defect to improve significantly.  Expecting things to change is a sure way to miss the target because when you discover that nothing changed, you will have less time to apply corrective actions.

Nothing Goes to Plan

It is my experience for software projects to always estimate optimistically.  There always seems to be more work than planned, there always seems to be more defects than planned, and there always seems to be more time required to complete assignments than planned.

The strength of an empirical model of process control is when things don’t go as planned.  When more defects are found than planned, analyzing the metrics tells management there is a problem early, the extent of the problem, and what precise options are available to put the project back on schedule.

Summary

To apply the 6-step process successfully, the following data requirements must be met. 

• The size of a deliverable must be estimated and measured.  The more granular the unit of size, the more control you will have over your schedule.  While I have my preference, any unit of size that is measurable works.
• Effort must be calculated from size by dividing it by a productivity rate.

Software metrics are often a controversial subject in the software industry.  I suspect because of the fear metrics will be misused to measure individuals: opposition to software metrics often revolves around the problem of individual productivity.  Software metrics, when used properly, is a tool for measuring the project not individuals; it’s a tool for controlling project deliverables, assessing and communicating status, and making better decisions.

It’s suggested that the web has changed everything.  Whatever requirements there were for delivering desktop applications, the requirements for delivering web applications has changed.

For the web, the thinking goes; delivery of new features to customers is paramount to remain competitive.   Maybe it’s true, but what’s the evidence?  My own experience with popular web sites does not support this conclusion.

Online Examples

I’ve been using Citibank online for over 15 years now, and I use it the same as I did when I first began banking over the web.  The banking features haven’t changed for a long time until recently.  While some features of the application could have been vastly improved, I never felt the urge to change my bank because of the web application. 

But even if Citibank’s competitors offered a vastly improved online experience, I’m unlikely to change my bank because of the web application.  The Citibank online experience satisfies my need already, though admittedly improvements could be made.  Plus, how much better can you improve the features to pay bills, transfer money, and view transactions?

I’ve been using Ebay as a seller and a sometimes buyer for 10 years; the features that I use haven’t changed much nor do they need to.  Ebay has the one most important feature that cannot be realized in software improvements: access to a large community of consumers. 

I use Amazon.com frequently, and the content is way more valuable to me than the functionality delivered and so is the price of the products offered.  The feature of purchasing products on Amazon.com hasn’t changed much, nor does it need to.  I go to Amazon.com not because of the software but because of the content, the price, and the reliable delivery.

I use “My Yahoo” as my home page, and it hasn’t changed much for a long while until recently.  I attempted to use their beta release and I switched back immediately as it wasn’t ready for prime time.  I’m too busy to devote effort to being a company’s beta tester.  I believe that’s another drawback of Agile: most customers don’t want to be testers.  But here again, the content is way more important to me than the functionality of “My Yahoo.”  

Recently, yahoo revamped their online email application.  Though the features have improved significantly and I’d even say it’s good, I still don’t use web email applications.  I prefer outlook.  I’m not sure why, I just do.

My former company used Concur for employee expense submissions and management. Its first release was a huge improvement over the manual systems of the past.  I used it for a number of years while I was there.  I don’t remember it changing much or at all over those years, but it didn’t need to.  Sure there could have been some UI improvements that would have made it much easier to use, but even if a competitor had a better interface, it would have had to offer something revolutionary to motivate change as the current experience with Concur was invaluable.

I use WordPress for this blog.  I love the application.  They recently released new software, and I promptly upgraded.  I like it, I think it’s better than the older software, but really, I would be satisfied with the old version if they hadn’t offered anything new.  The two most important features for blogging WordPress satisfied long ago.  A blog using WordPress is entirely customizable, and it offers an easy interface for managing content.

What Drives Consumers?

Consumers don’t readily upgrade or replace old products because they have new and improved features.  They replace old products because they have a new need or the upgrade product solves and old problem.  Think about how often you replace physical products in your home because the companies offer products with new and improved features.  How often are you willing to replace your refrigerator, television, stereo, iron, toaster, car, dishwasher, washing machine, stove, and microwave oven, to name a few common home products?  Most of these products are replaced because they are broken, no longer work reliably and/or as well as when they were new.

Summary

The web seemingly changes the economics for companies offering new/improved features because the web has lowered the barriers to delivering changes frequently.  Because the barriers to delivering new/improved features to the consumer are lower, does that mean we should deliver new/improved features more frequently?  What if the software features aren’t the business drivers for your web business?   What’s more valuable to an Amazon.com customer, a new book for sale or a new software feature on the web site?  The danger is that delivering software enhancements frequently gives the illusion that you are delivering more customer value when in reality it’s not the most important need for your customer(s).

The web has changed everything.  The web moved the focus from functionality to data.  Changing software features are not the most important change; it’s changing content that is king on the web.  It’s what I struggle with on my blog and most bloggers struggle with. Do we still need new and improving features on the web? Of course we do, but rate of changing features isn’t nearly as important as it’s argued to be.  Having the right new set of features timely is where the focus needs to be over having any new features quickly. That’s what will win on the web and on the desktop.  Identifying and delivering real customer value is never easy, and it doesn’t get easier with smaller more frequent deliveries.

“I’d rather drive a Pinto that works every day than a Lexus that always breaks down,” said an unsatisfied customer of a company that delivered a product plagued with quality problems.  For him leading edge was less important than quality, or to put in other way, leading edge features that don’t work is like not having the features in the first place.  That’s true for most customers. 

It appears that the software development community believes that customers have a high tolerance for product defects.  How else can you explain the low quality that we often experience in the software products that we purchase?   I’ve often written about my experience with video editing software where it was difficult to find a product that could compile an edited sequence successfully.  Essentially, many of the video editing products at the time could not reliably complete the primary function for which people purchased them: compile sequences to a single AVI file.

Consumer Product Experience

What if you experienced similar quality in other products that you purchase?  What if you purchased a toaster that always burnt the toast?  What if you purchased a lawn mower that was difficult to start?  What if you purchased a car that frequently broke down?  What if you purchased new furniture with an obvious blemish?  Most of us would find this level of quality unacceptable unless, maybe they are steeply discounted.  But not in all these cases.  If you always get burnt toast, you’d want your money back even if the workaround is to use a timer and manually eject the toast when enough time has elapsed.  It sounds preposterous to even suggest that workaround is acceptable, but we often ask customers of our applications to accept similar preposterous workarounds for our software defects.

Now what would be your impression of the engineering team if in addition to all these defects, there are obvious misspellings and grammar errors in the on screen text, error messages, and product labels?  I bet your reaction would be similar to mine: of course the product doesn’t work, these people are incompetent.  They can’t even spell or write good English.

Workaround Defects

During the heat of a fast paced release, there is often pressure to defer fixing defects to future releases.  If the defect has some functional workaround, we often rightly lower the defect’s severity.  Defects with no workaround are obviously more severe than defects with a workaround.  However, this becomes dangerous when it’s argued that it is acceptable to release the product with broken features having a workaround. 

To bring some more clarity to the decision to release with or without the fix, it helps to draw some analogies to problems in other products that we use.  For example, if we have a feature in our software product that sporadically does not function and the workaround is to try it a second or third time to get it to work, would we find similar behavior in our new car to be acceptable?  If we purchase a new car and the radio would sporadically turn on or turn off when you press the power button and the workaround is to smack the radio hard once between the volume control and the tuner control, would you find that acceptable?  You’d want it fixed, and you’d be darn mad, right?  If we’d find the analogous workaround unacceptable in other products that we purchase, chances are that our customers will find it unacceptable in our software products too.

Cosmetic Defects

Another source of deferred defects is cosmetic defects.  Maybe a button isn’t in the right place, a message box has misspellings, or an edit control is too short.  Often these are rightly assigned low severity.  None of these prevents the functionality from working, but they make the product look bad.   If we release with any one of these defects, we likely would get away with it.  What if there were a thousand of these low severity defects?  Can you still safely release?

Let’s make the automobile analogy again.  Let’s assume you go to the dealer to pick up your new vehicle.  You walk around the car to inspect it making sure there aren’t any defects that would prevent you from driving the car off the premises.  As you walk around, you spot a minor surface scratch. Would you still drive off with the car?  Most people would, but I suppose a very small minority would not.  Now what if there were a large number of these surface scratches all over the vehicle and when you look at the vehicle from differing angles, the scratches noticeably detract from the new car finish and make it look unsightly.  Would you still drive the care off the premises? 

Leave a Good Impression

The release date nears, and you have to decide whether to slip the date or release as planned.  The team decides to release as planned with all the open cosmetic defects unaddressed and all the open feature defects with workaround solutions unaddressed.   If your customers rarely discover these problems, then you made the right decision to release.  On the other hand, if your customers are discovering problems frequently and becoming frustrated with using your product, then you made the wrong decision.  What impression do you think your customers will have of your team when in addition to all the defects preventing them from using your product successfully, the product is littered with many spelling and grammar errors?  Does the word professional come to mind?

Quality Matters

I read it in the blogs.  Many writing about software practices believe that quality is a trade off – especially if we are developing leading edge technologies, but really it isn’t.  Do you believe the astronauts of the space shuttle, a leading edge technology, have high expectations for quality?  How about the taxpayers of the space shuttle who are shouldering the funding for the program?  Do they find it acceptable to see their hard earned tax dollars destroyed in a disaster along with the precious lives of the shuttle astronauts?  While it’s difficult to prevent all disasters, we are more likely to have disasters when we have a casual attitude towards quality.

Even if your product reliably performs as advertised, do you still want to leave many cosmetic defects open in the release?  If there are enough cosmetic defects, it’s still going to leave your customers with a negative impression of your company and the team building the product.  What is the image of the company and the team that you want to communicate?  If the image you want to communicate is a company that has top talent and aims to deliver high quality and the best user experience, you better fix most of those cosmetic defects.  If you have lesser objectives, well, you can leave most of those cosmetic defects present.

Summary

As a consumer I value high quality.  When a company develops a reputation for high quality, I am more likely to purchase their products, and once I own their product, it’s difficult for me to switch to a competitor.  I can think of a number of once famous leading edge technology companies that failed to deliver leading edge quality.  Most of those companies that I’m thinking about right now are no longer in business, or they are struggling to survive. I was never a repeat customer of any of their products.  I often replaced their products quickly with a competitor’s.

When you are prepared to release with defects having a workaround, put yourself in the customers shoes; ask yourself, would I be satisfied with that workaround?  Would I be satisfied with an analogous workaround in any other consumer product?  When you are prepared to release with many cosmetic defects, ask yourself, what image would I have of the team and company if I were a user of this product?  How would I react with analogous cosmetic defects in any other consumer product?  Chances are your customers will react similar to you.  It’s best you fix them if you want to keep that customer

In software, where high quality continues to be uncommon, high quality is a competitive advantage.  Yes, delivering high quality takes time, but in my experience, low quality always takes longer and costs more.  As I see it, focusing on quality is a win-win strategy: it saves time and money, and it creates a positive reputation for your products and your company. Plus, your customers will be loyal for it.

Why does it take so long to deliver software products? Many stakeholders ask this question during the course of a software development project.  It’s interesting when the developers ask this question because they know what it takes to implement the functionality in software, but in asking that question, some in the organization often fail to appreciate what it takes to deliver a commercial product to market.  From one perspective, they are correct; it shouldn’t take so long, but from a different perspective, the long duration is understandable — even if undesirable.  I’d like to explore the perspectives that explain these differences using a hypothetical delivery.

Example Requirements

You are asked to build an application that will copy files created for the day from your systems to your customer’s systems.  The files need to be converted into a format acceptable to the customer’s application.  Every night the files generated for the day are copied using the FTP protocol to the customer’s FTP server.  The transfer must complete within two hours.  The communication from your systems to the customer’s will be via VPN over the internet.

Why So Long

This should take a couple of weeks at most — maybe even a couple of days. To convert the files, identify an off the shelf library that will convert between the two file types, and to transfer the files, also identify an off the shelf FTP library to transmit the files.  If there’s good public domain software, use that.  To complete the application, identify the files to transfer, call the method(s) to convert the file, and then call the method(s) to transmit the file. Repeat this until all the files are transmitted.

This is reasonably simple.  It shouldn’t take too long to write.  So why may it take a couple of months to deliver this application?  A college student could probably complete this application in a day.

Why It Takes Longer

The algorithm above works in the ideal case, but errors can arise that need to be handled gracefully.   Connecting to the FTP server may fail.  There may be errors in transmission.  Maybe there’s not enough space left on the destination server to transmit all the files.  These are a few of the possible errors that can surface during operation.  At a minimum, the application needs to detect these errors and report on them.

The entire operation needs to complete in two hours.  This requires that the algorithms to convert the files are efficient.  Some analysis is required to assess whether the third party libraries will be efficient enough to complete the work in time.  Depending on the findings, there may be more design work required to meet the applications performance requirements.

The application has to be supported when it goes operational.  Documentation needs to be created for the operations staff to support the application when it goes live.   When errors happen, the operations staff needs to know the procedures for identifying and correcting the various types of errors.  This is necessary if the developers don’t want to be called in the middle of the night every time the application fails.

There’s a dependency to address for this application before it can go operational.  A VPN connection needs to be configured between your site and the customer’s site.  While this isn’t a difficult configuration to set, the network operations team has a large queue of work, and it will take some time for them to get to this task.

The third party libraries need to be purchased.  This requires identifying and evaluating the products, submitting a purchase requisition, placing the order, and waiting for the software to be delivered.   Just these steps alone can take a couple of weeks.  If the software is unavailable for download, this will be a dependency since coding cannot complete until the software is in your hands and installed.

End-to-end testing needs to complete with the customer’s application that consumes the files transferred.  In the ideal case, it’s a day or less of work if everything works well, but if not, analysis and troubleshooting are required to find and correct the problems.

Summary

Too often when stakeholders ask the question, why it takes so long, they naively look at the feature to deliver and rightly conclude it can be completed quickly.  However, there are more requirements to address than the simple algorithm that delivers the primary functionality.  Not only does the application need to work in the ideal case, it needs to handle error conditions correctly.  Performance requirements must be addressed, and depending on the variables, this can add significant development overhead.  There are support requirements and network configuration requirements, which contribute additional overhead to the assignment, as well. 

Many of the requirements that add to the project duration aren’t visible to the stakeholders asking the question, and consequently, they aren’t calculated in their optimistic view of when the assignment should finish.  So if you are still asking why it takes so long, it’s only and FTP application.  What aforementioned aspects of this assignment would you leave out?  Usually, there’s a good reason it takes longer than we’d like and even expect.

Achieving Agile goals of delivering faster and responding to change quickly is best achieved by designing an architecture that leverages those goals, I reasoned in an essay titled “Agile Isn’t a Process“.  Being agile is about leveraging the entire organization and business partners to deliver solutions to your customers. Agility is best achieved when solutions require less IT involvement, not more. 

While having a good development process is important and necessary, the Agile process is not well suited to developing agile architectures; an emergent, iterative approach does not offer much when the requirements for solving the problem require an investment in up-front analysis and design. Some would call the investment big, but I would call it proper.  In this essay, I’d like to offer a real world example to support that original thesis.

Criticisms

It’s interesting to read some of the criticisms of the article. One reader suggested that an investment in up-front analysis and design results in overly complicated solutions.  George Crews commented, “It has been my experience that taking a long time to architect software only guarantees complex software, not software that will actually work.” 

It’s a common opinion.  I actually agree with this comment if taking a long time is evidence of having an untalented team working on the problem.  I have found the weaker performers always seem to take longer and develop more complicated solutions. On the other hand, if the team is talented, taking a long time is evidence of a difficult problem, and difficult problems require an investment in time for even talented people to solve well and simply.

Another common criticism is that short iterations reduce risk. Well, it depends.  If the iteration is designed to deliver less, then sure, risk is reduced.  On the other hand, if the short iteration forces the developers to cut corners to deliver within the iteration, risk is increased as requirements are missed and error conditions aren’t considered in the interest of delivering something quickly.

It’s also suggested in some blog articles that investing time in a simple but thoughtful architecture compromises present requirements. Big Design Up Front (BDUF) steals time from delivering on the more pressing present requirements.  It’s said that present priorities require precedence over future needs, and therefore, it’s suggested to plan just enough for the future.  However, if you expect the product that you are building to have longevity, then the future is a present requirement that should not be ignored.

Of course, the future has to be balanced with the present; there is no formula for this.  Arriving at the right balance between present and future requirements is an art; it is situational (sometimes future requirements have higher priority and sometimes they don’t), and striking the right balance is usually a sign of a vibrant team, a team where contrasting ideas and opinions are plentiful

There’s another criticism in the present versus future argument. The position is that analyzing and implementing solutions that anticipate the future requires more time. My experience doesn’t support this position.  Yes, analyzing and designing a good solution often takes more time to deliver that first working build, but the time lost in getting to that first build is often compensated in delivering the overall solution faster.

Another suggested reason for delivering a working solution faster is that it reduces risk since working code proves out the design.  Risk is increased when the production of working code takes longer, since a flawed solution will be found late in the release cycle, where rework will impact the schedule.  This position is specious.  Up-front analysis and design is a risk mitigation strategy; it doesn’t increase risk.  I’ve never seen a design where I could not determine whether it would be successful or not.

BDUF Example

In 1991, I was assigned to develop the solution to realize the requirement to internationalize our product.  This product was known as Reuters Terminal.  It was essentially a financial browser for navigating and displaying Reuters’ real-time financial news and information.   Reuters financial network disseminates financial news and information from every market in the world to every financial data center in the world.  Consequently having a product that was in the native language of the local market would be a competitive advantage – especially at this time when internationalization support in the Windows operating system was just being conceived.

The culture of this team was to do what is pejoratively called today Big Design Up Front, but to us, it seemed like the sensible way to design a good solution and to deliver it to market quickly.  Project initiation to release was approximately six to nine months, and there were many requirements in addition to the internationalization requirements that were delivered in this product release. 

Design

The architecture was a 3-tier architecture:  The top tier is the application requiring the internationalization support.  The middle tier abstracts the details of each language instance from the application, and the bottom tier is a language driver that captures all the details and nuances of rendering an instance of any particular language for the application. The application resources are linked into the language drivers.  Essentially all the Windows APIs for rendering text, dialogs, and menus are replaced in the middle tier.  Below is a pictorial representation of the architecture.

Benefits

While the first build may have taken longer than a build if this was implemented with an Agile approach, this approach has considerable advantages over the impetuous need to have a quick first build.  The advantages were. 

• Increases implemenation parallelism.  After the interfaces for the language API layer and language driver layer are designed, the implementation can be accomplished in parallel with one or more developers working in the application layer, one developer working in the language layer, and one or more developers working in the language driver layer.  The implementation is highly scalable.
• Anticipates future requirements. The application is to be customized for many languages, and this approach permits support for many languages without obfuscating the code with many macros to selectively link the application for any particular language.  It permits new languages to be released without releasing new versions of the application.  And it permits internationalization to be outsourced without much involvement from the core application team.
• Provides the capability to switch between languages at run time.  With this architecture the application can switch between languages at run time.  This is in part possible because of the architecture.  The application works with the middle tier only, and switching between languages is simply having the middle tier dynamically link to one of the language drivers.  Plus, there’s essentially very little extra code to support that feature.  It requires the middle tier to activate a different language driver, and the application forces each of the windows on display to repaint.
• Makes software configuration management practices easier.  Since at the application layer the code behaves the same no matter the language, there is no need to merge code while additional languages are in the process of being supported, and since support for new languages are delivered independently of the application, there is no need to manage and control many different versions of the application.

Once the internationalization architecture was completed in that first release, the architecture was in place for the lifetime of the application.  Re-factoring is rare when problems are analyzed well.

Agile Example

An Agile team has one primary goal: deliver working code in a short iteration.  If these are my goals, I would immediately begin tackling one aspect of the user interface that I can readily begin to internationalize.  Perhaps, one would start reworking all the text writing code in the application to display multilingual.  To begin seeing results quickly, the easiest approach would be to statically link a different resource file.  Then, rewrite the application code wherever LoadString and TextOut are used.   To deliver the solution quicker, one or more developers would be assigned different application modules to internationalize.  While this approach would deliver working code fast, it has some disadvantages. 

• Code is obfuscated with macros.  To update the application code for each language, the logic for each language would be selectively compiled with macros to enable the compilation of the code for each specific language.  When enough languages are supported, the language code overwhelms the primary logic in those procedures.
• Each module would likely solve the problem a little differently.  By implementing the solution inline, there is more variability in the implementation when multiple programmers are working on each module.  Even if the same developer were to fully implement the entire solution, there is likely to be variability in the algorithm as the developer converts each module. This makes the product more difficult to maintain.
• Quality is lower. Since the internationalization is completed inline rather than via a one-for-one Windows API replacement, a defect in the inline code may need to be fixed in every place that it’s been re-engineered, but a defect in the API only needs to be analyzed and fixed once.  Additionally, there’s more of an opportunity to introduce a different defect in every place that the inline code is touched.
• The implementation is less scalable.  It’s difficult, if not impossible, to outsource the internationalization development as code merging would be a requirement.  There is even less ability to parallelize development when the same file needs to be re-engineered for each language to support.
• Adding new languages is harder.  As new languages are supported, there will be constant merging of files with work going on to support other features.
• Configuration management is more complicated. Each supported language requires a separate build of the application, and therefore, a separate executable is required for release.  Also, a bug in one language may not be present in another language, so the code base will be changing out of synch with the executables deployed.

This is certainly a viable solution with distinct and troubling disadvantages for moving the code base to the future and responding quickly to changing requirements.  Unnaturally forcing a short iteration has disadvantages.  We used to have or own pejorative name for this: BFM, Brute Force Method.  Shortchanging up front analysis and design forces a BFM solution.

Summary

When I was implementing this solution with up-front analysis and design, we didn’t use the words Big Design Up Front.  We called it good analysis and design.  Sure there were engineers who spent too long in analysis and design and arrived at overly complicated solutions, but this had nothing to do with the approach: it had everything to do with the personality of the individual solving the problem.  Placing the blame on BDUF is a case of mistaken identity.

Proper up front analysis and design is required, and yes, sometimes it takes longer than we’d all like it to take.  However, when it’s done well, architectures are less complex, not more; quality is higher; time to market is faster, and products and teams are more agile.  Proper architectures leverage the resources of the entire organization to deliver solutions to customers.

Early in my career I had the good fortune to work under a management team that valued producing high quality applications – ones that rarely, if ever, crash under continual load.   The director of the department believed, not only in words but in action, that a defect discovered in the field is significantly more costly to fix than one discovered in development.  We lived by that rule, and we all believed in it.  

For this product, we did not have a sustaining engineering team to fix and deliver patches to the field because it was simply not needed. Over the course of the seven years of working on this team through multiple product deliveries, I recall only two or three issues from the field that required an engineering fix.   Now think of all the features you could deliver by investing the money saved in a sustaining engineering team into new feature development. That’s what we did.

I’d like to describe the six practices required to produce and release stable applications.  The level of stability that you achieve in your products will correlate to the degree to which you commit to these practices.  The strategy by itself isn’t enough to achieve great results.  Great results require discipline, persistence, and a strong will, and most importantly, it requires a leader and a team that values quality above all other criteria.

Quality is Everyone’s Responsibility

While the quality assurance team has quality in its name, quality is everyone’s responsibility.  On this team we didn’t have a quality engineering team.  Developers wrote and executed the performance tests, regression tests, integration tests, and acceptance test plans.  While there was no mistaking that quality was everyone’s responsibility on this team, it isn’t necessary to have the developers perform double duty as the quality team, but they do need to be involved in the test activities. 

Unit testing is something that must stay with the developers, and it should not be shortchanged to secure a delivery.  The developers should have responsibility in reviewing the test designs produced by the quality team, and they should be involved in designing and executing the performance tests.  If the need arises, developers should be available to pitch in to support the test execution effort.  Developers always should be available and open to questions from the testers.  If the engineering culture is that developers only write code, the team isn’t prepared to produce higher levels of quality.

Practice Defensive Programming

This approach was popular when I started working in the field, but it seems to have lost its luster.  I guess because it appears to take more time.  Defensive programming is about writing code to proactively detect error conditions and respond to them in sensible ways to prevent the program from failing or to have the program fail gracefully.  There are two primary benefits to defensive programming:  the developers become more focused on preventing errors, and when errors do surface, they can be solved quickly as the error is detected closer to where the error materialized.

Instrument the code

Have a debug version of your code with instrumentation to facilitate debugging when error conditions are detected.  This improves the turnaround time for fixing defects when they do occur.  One of my favorite practices is liberal use of the ASSERT Macro in C.  This practice was made popular in the 90′s by Steve Maguire in his book titled, “Writing Solid Code.” Use the ASSERT macro to validate parameters passed into procedures and functions, and use it to assert the conditions upon exiting a procedure or function.  If you are detecting error conditions at the earliest instance in which they materialize, you will dramatically improve your team’s ability to find the root cause quickly. 

ASSERTS differ from defensive programming in two ways: they are only present in debug versions of the application, and they do not take corrective action; instead, they break into the debugger when the error condition is detected.

Do not limit instrumentation to ASSERT macros only.  Some other good practices are to have execution tracing logic, initializing memory when allocated, and object validation logic to name a few others.

Soak Test the Application

Soak testing is the key practice.  Soak testing is the practice of exercising the application under automated user control.  The goal of the automated testing is to put the application under continuous user load to surface problems that would only surface after repeated usage.  It is best if the user input is randomly generated, but it isn’t required to have good results.  

If the application processes input other than user input, it is necessary to have simulators to generate that input and to pump that into the application continuously while the application is under automated script control.   At least every morning the applications should be surveyed for error conditions.   These are some of the candidate error conditions to monitor:

1. Memory Leaks – is application memory usage continually growing?
2. Secondary storage leaks – is the application consumption of secondary storage continually growing?
3. File handle leaks – are the open file handles continually growing?
4. Lock ups – is the application still responding to user input or does it appear to be locked/hung?
5. Crashes – has the application committed an operating system protection violation?

If any of these conditions are detected, they should be analyzed and fixed immediately. Keeping the application running continuously under automated script control has precedence over all other issues.  Once the issue is fixed, a new build is released to the lab to undergo soak testing. 

Performance Test to Failure

Many teams will execute performance tests to validate that the performance objectives of the system have been achieved and stop there, but it’s not enough to release a stable application.  There is really no telling how your customers will finally use your application.  Performance test the application beyond the performance requirements until it fails. You may be surprised to find that the application can exceed the performance objectives significantly by addressing some minor issues revealed when the application is stressed beyond the requirements.

The final objective is that when the user exceeds the performance specification, the system handles the condition gracefully. It should not crash, and it should inform the user that performance has degraded, and that calculations are unreliable while it is operating in this overload state.  When the load settles within the bounds of the specification, the application should recover and begin to perform normally.

Have Stability Release Criteria

The release decision has to be objective, has to be reflective of the test results, and has to support the goal of releasing a stable product.   Here are some recommended stability release criteria.

1. Identify the length of time for the application to run uninterrupted under soak testing.  I like to set the target at a multiple of the expected user usage.  For example, if I’m testing a professional trading application, the work week is 5 business days or one week.  At a minimum, the application should run uninterrupted for 5 straight days, but I would try to have that application successfully run for 30 days or one month under soak testing.  The release build should have at least achieved the minimum criteria.
2. Have no known resource leaks in the application at release time: dynamic memory, secondary storage, file handles, or any other limited resource used by the application.
3. Have no known crashes at release time.

Summary

It is my belief that releasing a high quality product is the easiest way to differentiate your product, and low or sub par quality is the surest way to lose a customer forever.  It’s difficult to unglue a costumer from a reliable product even if it does not have all the bells and whistles of competitive products, but of course, it must have the essential features to be desirable.  We only have to look at the American auto industry for a great example of how low quality can destroy a once great American industry.

I recall a number of years back when I began to develop an interest in video editing on the PC.  After evaluating a number of products, I settled on Adobe Premiere.  While the quality of Adobe Premier has improved over time, the initial version that I worked with could not complete the job that it was advertised to do, and it was true for all the competitive products that I evaluated at the time.  I only settled on Adobe Premiere because it came bundled with the Matrox video capture card that I purchased: RT X10. 

My video editing needs aren’t demanding.  I didn’t need many bells and whistles.  All I needed was the ability to capture, to splice multiple clips and stills, to overlay audio, to edit some titles, and to insert transitions.  As an amateur editor, it couldn’t reliably satisfy my limited demands.  If I could have found a less feature laden product with higher quality at a premium price, I would have purchased it without hesitation.  I have no doubt if there was a competitor offering a competitive product with a reputation for high quality, the product would have been able to acquire significant market share.

Quality is a differentiator.  Grab the low hanging fruit; put these practices to use in your own projects, build stable applications, and you will differentiate your products from the competition.

Have you ever wondered why software process has yet to flourish in the software industry?  Why, after many decades of industry growth, there is no consensus on a process methodology or even best practices?  Why specious Agile approaches have captured significant mindshare in the software community?  Why when you mention the words software process, software professionals cringe?

In other industries process innovation has been the contributing factor in the explosive growth.  Where would the industrial revolution be without the innovation of assembly line processes?  Where would the auto industry be?  A predictable, repeatable process has been the cornerstone of innovation and quality for many successful industries.  Did you ever wonder why a hamburger and french-fries at McDonalds taste the same no matter where in the world you purchase it?  It’s all about process.

It’s an enigma to me and others who have had great success with software process that we continue to debate the need for formal process in the software industry.  Yet the software industry continues unabated with a record of late deliveries, budget overruns, missed expectations, and low quality.  In this article, I’d like to explore the factors that contribute to software process adoption failure.

Undisciplined

People are undisciplined by nature.  It requires effort to be disciplined, and you usually need to be disciplined for things you need to do but find hard to do.  For example, how many of us can stick with a regular exercise or diet program?  Both activities are great for our health. It’s difficult to argue against the need for discipline to be successful in those healthful endeavors, yet most of us find it difficult to stay with an exercise or diet routine for long.  Similarly, software process is difficult to stick with, though it benefits our projects.  Without discipline, software process is sure to decay and ultimately to fail.  Reward, hire, and promote people who demonstrate discipline.  The most successful people demonstrate the highest degree of discipline.

Bureaucracy

Many Software Processes are overly bureaucratic. I’ve noticed a strange phenomenon when people are given the assignment to develop a process area: they behave as if they are writing a Ph.D. thesis.  They create an elaborate and complicated solution when a simpler but appropriate solution would suffice.  My former supervisor would often quote, “Process should be like a woman’s skirt: short enough to be interesting, long enough to cover the subject.”  Process should be lean and mean.  Bureaucracy is an unwelcome friend of process.  This is a behavior that one needs to be continually vigilant in preventing to develop process that succeeds.

Ineffective Process

Some processes, maybe even many, aren’t good.  Just because you have invested a great deal of time developing and deploying your process, it does not mean it is effective or good – even if you were successful in achieving an audited process level.  If you are not getting the benefits from your process that you expected, think about reassessing whether you’ve developed and deployed an effective process. Most importantly, fix it.

Management Priority

Management needs to show by example that they believe process is important. Employees take serious what management demonstrates to be important.  As identified earlier, process requires discipline from the team.  If management isn’t making sure that the process is being followed, the discipline will wane.  If you are a manager and you don’t believe in process, don’t even start.   You’re only wasting the organization’s money and time.

Management Commitment

It takes commitment to be successful with process.  Gaining acceptance is often challenging because people find it difficult to change their practices – even if the change promises to improve their working conditions.  Don’t expect a good process to be introduced in less than 18 months:  normally teams are juggling their current project commitments with activities to deploy a new process.  A comfortable measured pace is what’s required.  After the project is rolled out and fully deployed, it takes continued commitment to sustain the process.   This is very much like starting an exercise program to lose weight.  It takes time before you start seeing results, and once you reach your goal, it takes continued effort to maintain the weight loss.  I would guess that about 50% of organizations that failed, never reached their process goal, and the other half quit during maintenance.

Misused

Often time software teams find themselves committing to extremely aggressive deliveries, and they misuse process as a way of slowing the business groups down.  When teams do this, you’ll hear comments like the process says we don’t deliver estimates until the requirements have been signed off or similar types of arguments that delay the development team from making a commitment.  This is the surest way to undermine the business group’s support for process in your organization.  When the business group doesn’t get the cooperation and support that it deserves, they will eventually withdraw their support for your process.  Not only will this behavior result in the abandonment of the process, but it will impede the software group’s ability to manage the work load more effectively, as a good process is the only way to achieve that.

Overemphasize Process Levels

When organizations embark on process initiatives, management will often set a deployment goal.  For teams using CMMI as the model, the goal will be to reach level 2 by a given date.  This is the wrong emphasis as reaching a process level signifies nothing.  Like any certification, reaching a process level only signifies that you’ve met the requirements of the discipline.  It has loose correlation to process effectiveness. The reason to deploy process is to improve your organization.  Have measures for demonstrating improvement, and let those measures be the primary goal. 

I don’t intend to suggest there is no value in achieving process level maturity.  It’s the emphasis that concerns me.  It’s easy for teams to game the system and reach a maturity level when they really aren’t following much of the process that they present to the audit committee.  If the emphasis is on tangible results, there is less reason to game the system.

Dedicated Process Groups

It’s natural for organizations to embark on a process initiative by creating a process group staffed with full time process engineers.  While the effort required to build and roll out process in an organization is a great deal of work, too often this is the least effective approach to rolling out software processes for many reasons.  There are two that make it a particularly incorrect approach.  One is that often dedicated process teams are staffed with people who have never been successful in managing the delivery of a product themselves.  On paper delivering software looks like a simple endeavor, but it’s impossible to get a feel for the nuances of leading and managing a team from a book.  You need to experience it.  Weak, inexperienced engineers develop unproductive processes. The second drawback of a dedicated process group is that the process group’s self-esteem is tied to rolling out process, so they tend to document and develop many procedures to be followed.  Before you know it, you have layers upon layers of bureaucracy for delivering a single line of code.  It’s better, yet more challenging, to have process teams staffed with employees who have a part time responsibility for creating and deploying the process.

Poor Process Audits

As identified earlier, good process requires discipline.  Audits are necessary to enforce discipline. However, there are some pitfalls.  Auditors may work with the people they audit everyday, and in this case they may find it uncomfortable writing up a negative audit report.  One way to combat that is to have formal audit objectives with specific pass or fail criteria.  This removes the auditor from making a subjective call on the audit.  The only question the auditor should set out to answer is, is the team following the documented process?

Doesn’t Promote Change

The only ideal process is a process that changes.  Like a software release, the first version of your process will be buggy.  You’ll need to tweak it, and maybe even overhaul parts of it.  As the team becomes more experienced with process, they will find the practices that they originally thought would be effective can be improved. The process needs to encourage change, and it needs to be easy to change.  People will do what works, so it’s important to make it easy to change; otherwise, the documented process will not reflect what people actually do.

Fails to Demonstrate Benefits

Teams embarking on process often overlook ensuring that the process demonstrates benefits to the organization.  The business benefits must be one or more of the following: improved time to market, increased productivity, reduced costs, improved quality, reliable and predictable commitments, or improved customer satisfaction.  If there is no demonstrable benefit, why continue to do it, and why should management continue to sponsor it?

Been There Before

For the seasoned employee, chances are they’ve been through this exercise one or more times in their career, and usually the experience isn’t positive: management removes its sponsorship as things were actually starting to improve. Consequently, people are reluctant to invest in something that management won’t sustain. People become cynical about these initiatives, and they come to know if they hold management off long enough, they will abandon the initiative.

Companies Are Too Dynamic

The organization structure in companies today is too dynamic.  Way before a process has a chance to be institutionalized, the key sponsors are off to new assignments, or the teams are broken up and reassembled offshore.   It’s important to have organizational stability to nurture culture change in an organization.  This one is a tough one to solve.

Business Groups Are Uncomfortable

It’s difficult to get the business groups on board with process.  They are often satisfied  to manage release requirements informally.  They often fail to appreciate the need to prioritize requirements and create product roadmaps.  They struggle to identify release commitments with anything less than everything.  When you attempt to socialize the product group into the new software process, they feel constrained, and when they think the process is the reason they aren’t getting what they feel they need, they kill the process initiative.

Summary

There are many challenges to overcome when rolling out software process in a company.  Adopting and rolling out process shares many characteristics with keeping to a diet program.  They’re both good for you, they require discipline to be successful, there are many forces working to derail you, and there are many fad programs.  The current process fad is the Agile methodologies.  Like the fad diet program that promises fast weight loss with no discipline, Agile practices promise faster releases, higher quality, and more satisfied customers with less discipline.   And like the dieter eager for a quick, painless weight loss program, the software community is eager for a silver bullet.

A former boss of mine had a sign hanging on his wall that read “good software takes time.”   Likewise, good process takes time: time to develop, time to rollout, time to perfect and time to institutionalize.  It requires discipline, commitment, experience, and savvy to develop and maintain.  If you are embarking on a software process initiative or you are dissatisfied with the progress of your current initiative, it would be worthwhile to review the challenges identified in this article and develop a mitigation strategy for the items you believe may impact your program.  For those who are prepared and eager for the challenge, the rewards of successful process implementation are great.

Further Reading

• CMMI(R): Guidelines for Process Integration and Product Improvement (2nd Edition) (The SEI Series in Software Engineering)
• Real Process Improvement Using the CMMI
• Agile Project Management with Scrum (Microsoft Professional)
• Agile Software Development, Principles, Patterns, and Practices
• Extreme Programming Explained: Embrace Change (2nd Edition) (The XP Series)
• Extreme Programming Installed (The XP Series)
• Systematic Process Improvement Using ISO 9001:2000 and CMMI(sm)
• ISO 9001, The Standard Interpretation
• ISO 9001 Survival Guide, Third Edition

“Another 30 defects uncovered yesterday,” reports the QA Lead.  “Ten defects were fixed,” reports the Tech. Lead.  Taken alone these are alarming statistics, and if they persist long enough, the release date would certainly be in jeopardy.

Many software teams struggle through the QA phase with great anxiety as they normally work through the phase without a compass: nothing to tell them whether they are on track or not.  How many defects are left?  Will we be able to fix all the defects before the release date?  Will we find all the defects before the release date?  These are a few of the questions that teams have anxiety about.  It’s only until a few weeks before the planned release date that they come to grips with the realization that they are in trouble.

Measure, Graph and Track Output

Over the years, I’ve developed some tracking techniques that have helped me to control the QA phase of the software development life cycle.  I like to record, track, and graph some key statistics that I use to measure how well the project is progressing to schedule.  The data points are recorded and graphed daily in an Excel spreadsheet and maintained weekly on a historical basis.  Basically, I record the current week’s data daily in the same cell until the end of the week when the last value persists and begin the process in a new cell for the next week.

Based on the behavior of the trends, I make objective decisions for reporting status in the weekly status report.  Red means our fix rate is trending to deliver the project beyond the scheduled release date unless something is done to correct it.    Yellow means that our fix rate is trending off schedule, and if current trends persist, the project will deliver late.  Green, of course, means that the project is delivering on schedule.   I’ll detail the specific criteria for determining status later in this article.

There are 7 variables that I track during the project:

1. Actual defects found as a line plot.
2. Weekly defects found as a bar graph plot.
3. Actual defects left-to-fix as a line plot.
4. Weekly defects fixed as a bar graph plot.
5. Expected defects to find as a straight line.
6. Expected defects left-to-fix as a straight line.
7. Backlog of open defects as a line plot.

To make this all work, during the estimation phase, I estimate the total number of defects to be found and fixed before the product is released.  (I’ll discuss how I estimate these in a future article).  Figure A is an example of a typical graph with all the data points recorded.  The features of the graph as described above are labeled with their corresponding numbers.

**Note, a single click on all graphs in this article enlarges them and then reduces them after they’ve been enlarged.

Figure A. Labeled Defect Tracking Graph

Notice the 2 straight lines that cross each other.  The line starting from zero and increasing over time is the plot of the expected defects to find trend and the line starting from approximately 2000 and decreasing over time until it reaches zero is the expected defects left-to-fix trend.  The reasoning behind the behavior of the lines is that when you start the project all the defects are left-to-fix and when the project completes, there are zero defects left-to-fix. Likewise, when the project starts, none of the defects have been found, and when the project completes, all the defects have been found.

The purpose of these two lines, what I call idealized lines, is to compare and to contrast them with the corresponding actual lines for found and left-to-fix over time.  The behavior of the actual line data plots in comparison to the idealized lines gives you a measure of how well your project is progressing to schedule, and it quantifies the degree that the project is either ahead or behind schedule visually and measurably.

Calculate Project Sigma

When the fix rate is tracking ahead of schedule, the actual defects left-to-fix line plots below the expected defects left-to-fix line, and when the fix rate is tracking behind schedule the actual defects left-to-fix line plots above the expected defects left-to-fix line.  Similarly, when the find rate is tracking ahead of schedule, the actual defects found line plots above the expected defects to find line, and when the find rate is tracking behind schedule, the actual defects found line plots below the expected defects to find line. 

To get a measure of the degree the lines are behind or ahead of schedule, move the end date of the expected lines forward or backward until the actual line plots with the trend nearly identical to the expected line.  Subtract the calculated end date from the original end date. This value is what I term Project Sigma.  It is negative when the project is tracking behind schedule, and it is positive when the project is tracking ahead of schedule.  The larger the absolute value of Project Sigma, the greater your project is ahead or behind schedule. 

Calculate Project Status 

By analyzing the behavior of the actual line plots against the expected line plots it is possible to calculate and report objective and accurate project status.   As the current date approaches the release date, the actual line plots should plot with a narrow band from the expected line plots; similarly, when the project is further away from the release date, the actual line plots can plot with a wide band from the expected line plots and still deliver to schedule.  This behavior reflects the lower defect find and fix productivity early in the project and increased productivity as the project approaches the release date. Given this behavior, the thresholds for reporting status are relaxed further away from the release date and then restricted as  the release approaches the release date.

I recommend to use two thresholds to calculate project status.  The thresholds in use are determined by where the current date falls with respect to the red line date.The project red line date is the date at the point of interesection of the 2 expected lines.   When a project crosses the red line date, it is in the end game where the project team is predominantly testing the application and fixing defects.  Before the  project crosses the red line date, relax the status reporting triggers and restrict the reporting triggers when the date crosses the red line date.  Only the left-to-fix lines are used to calculate project status as the defects left-to-fix  is normally the critical path for the project: you have to find the defects before you can fix them. 

Below are suggested thresholds for calculating status.   Table 1 identifies the triggers to use for reporting status before the red line date, and Table 2 identifies the triggers to use to report status after the red line date.  First, calculate Project Sigma as described earlier.  Assuming the project has crossed the red line date,  find the column where the value of Project Sigma meets the criteria in the header.  When Project Sigma is postive, the status is naturally green, and no table lookup is required.  When Project Sigma is negative, use the absolute value of Project Sigma to index the tables.

What’s important here is the methodology — not the absolute thresholds and values.  Play with this.  You may find that you need to identify three sets of threshold criteria instead of two as I have shown.   Or, the values for indexing into the tables may not be suitable for your corporate culture, so you may decide to tighten them or relax them.  Change anything about this to suit your requirements, but change them using some logical criteria.  Don’t randomly pick a date for which you would begin using a third set of status criteria.  Create an algorithm to identify that date based on the observed behavior of your projects.

Under Estimated Project Signal

One behavior that I’ve repeatedly observed is that when the actual left-to-fix and the actual found lines intersected prior to the expected line’s intersection, it has always indicated early that the initial project estimates are wrong and the project is underestimated. This behavior is natural as most projects are optimistically estimated or, as some would argue, aggressively estimated.  If you see that behavior on your own projects, it would be a good idea to re-estimate the project.  At a minimum re-evaluate the assumptions that you based your schedule upon to confirm if they are still valid.  Chances are things have changed and you haven’t reflected that in your schedule.  This example is illustrated in Figure B.

Figure B. Actual Left-to-Fix and Actual Found Lines Cross Before Expected Lines

Calculating a New Release Date 

If while tracking your project you’ve concluded that you will not deliver on schedule, this method of tracking becomes very valuable for projecting a new release date. To project a new release date when you’ve concluded the project is off the tracks, you first re-estimate the number of defects that you expect to find and fix in the project.  Plug those new values into your Excel spreadsheet. Then extend the expected left-to-fix line until the actual left-to-fix line plots below the expected left-to-fix line.   Provided you haven’t improved your capacity to fix defects, that date is your new release date.  Figure C illustrates a project that is behind schedule, and figure D illustrates the same project with the new projected release date identified using this technique.

Figure C.  Metrics are Trending Behind Schedule

Figure D.  New Delivery Date Calculated

Summary

Using metrics based on measured output is the most reliable method for tracking a project and assessing whether the project is progressing to schedule.  Once my projects enter the test and defect fixing phase, I rely solely on the metrics and techniques described here to report status to the team and management as to how the project is progressing to schedule.   There are essentially two benefits to this approach.  One, it takes the guess work out of assessing your project status with regard to delivering on the release date. With that you can report on your project status objectively and accurately.  Two, it provides for a quantifiable methodology for controlling the productivity of your project team.  Knowing actual find and fix rates, you can add people or request overtime to bring your project in on schedule when you’ve identified a delay.  In a future article, I will describe how to use this information to motivate and focus the team to deliver on the project objectives efficiently and expeditiously.  

I know once you begin using this tool, you will find it invaluable for managing your projects.  I’ve uploaded an example spreadsheet for you to begin using on your own projects (Example Spreadsheet).  It is populated with all the data and graphs used in this article.  Give it a try, and let me know how you make out, and I’ll be happy to answer any questions you have when posted as a comment to this article. I’m sure you will find this technique invaluable as everyone has who I’ve introduced this to in the past.   Good luck!