• 80% of the sales are closed by 20% of the salespeople.
• 80% of the code on a project is written by 20% of the staff.
• 80% of the processor execution time is spent in 20% of the processor’s instructions.
• 80% of your customers use only 20% of your application features.
• 80% of your defects are in 20% of your code.

Erik Peterson presents on the 80:20 rule at a Google Talk, which you can view on YouTube.com.  He explains how to use the 80:20 rule to improve the quality of your testing with the benefit of delivering high quality software products.

History

The 80:20 rule was first demonstrated by Joseph Juran.  The 80:20 rule says that 80% of the effects come from 20% of the causes.  It’s an indicative trend supported by observation and experimentation.

The principal was coined by Joseph Juran, a management consultant notedfor his focus on quality.  He named it after an Italian Economist, Vilfredo Pareto who observed that 80% of the wealth in Italy was owned 20% of the population.

IBM was able to use this insight with great effect when they found that 80% of the execution time was spent in 20% of the instructions.  With this information IBM optimized those 20% instructions giving them a competitive performance edge.  Apparently IBM treated this as a closely guarded secret.

What exactly is it?

Mr. Peterson explains that in Cricket the sweet spot on the bat is where the minimum effort maximizes return.  It’s that small area, the 20% area, on the bat that will catapult the ball the furthest with minimal effort from the batter. It’s what makes professional athletes as they are able to connect the ball with that small area of the bat more often than the amateur.  Identifying the sweet spot, that small area on the bat, is what Pareto Analysis is all about.

Pareto Analysis is the study of the main causes/classifications of something to link them to the properties in the results.  The objective is to establish the relationship between causes and results.

7 step process

A graph is used to perform Pareto Analysis.  There are 7 steps performed in the analysis to identify the most important causes.

1. Build a table listing the causes and their frequency as a percentage.
2. Arrange the rows in the decreasing order of importance of the causes (i.e, the most important cause first).
3. Add a cumulative percentage column to the table
4. Plot with causes on the x-axis and cumulative percentage on y-axis.
5. Join the above points to form a curve.
6. Plot (on the same graph) a bar graph with causes on the x-axis and percent frequency on the y-axis.
7. Draw line at 80% on y-axis parallel to x-axis. Then drop the line at the point of intersection with the curve on x-axis. This point on the x-axis separates the important causes (on the left) and trivial causes (on the right)

Example Behavior

How many Peanut characters did Charles Schulz introduce into his Peanuts cartoons? A survey was conducted where people where asked to name their favorite Peanut character.  The results of the survey unsurprisingly followed the classic 80:20 rule where 80% of the total favorite votes were for 20% of the characters: Snoopy, Charlie Brown, and Linus.  The answer to the beginning question is that there were 54 named characters in the cartoons.

Agile and the 80:20 Rule

Conceptually the Agile paradigm is based on the 80:20 where the objective is to deliver the most important features first. If those features are identified well, it is likely that only a small percentage of the universe of features will satisfy 80% of your customers.

I plan to discuss more about the Agile relationship in a future posting.

Testing and the 80:20 Rule

In 1976 Glenford Myers recommended writing test cases before you execute them, and he observed that defects cluster.   Based on this observation, he recommended pausing what you are doing and writing more tests for where you are finding bug clusters.  In addition he found the following:

• 80% of the defects come from 20% of the modules.
• 50% of the modules are defect free at system test.
• 10% of the modules are responsible for 90% of the system downtime.

Implications

1. Defects are where you are finding them. If you are finding defects in an area, there are more there.
2. Detailed tests are not required for all functionality.
3. Sweet spots are where a majority of the defects are. Find the sweet spots, develop more tests in those areas, and test them more.
4. 10% of the modules are responsible for 90% of system downtime.

As Beck would recommend, look for code smells.  Bugs are where you smell them.

How to Find Code Smells?

Unfortunately, you won’t find the Pareto relationships until you start testing.  It’s essentially a feedback loop, but there are ways to be proactive:

• Identify features/functionality by risk and develop more tests there.
• Concentrate your testing on new features or complicated areas.

As any QA professional knows, often developers aren’t very helpful with identifying these areas.  Erik recommends looking at the checkout records.  Where there are more checkouts, the risk of defects is likely higher.

Alternatively, focus on the areas the customers use most of the time and make them really usable.  Most defects will go unnoticed by customers where the user experience is superior.

Prioritize on Feature Use

Test where the users spend most of their time.  Microsoft apparently uses approach to deliver their products, and in my experience with great effect (see Microsoft’s CEO: 80-20 Rule Applies to Bugs, Not Just Features).   I rarely experience a defect in Microsoft’s desktop applications; the operating system is another story, however.

I would sometimes wonder how Microsoft could deliver such high quality in their office products.   In hindsight it’s quite obvious how they achieve it.  I suspect that I’m like most Office users; I use 5% of the functionality 98% of the time.

If they get that 5% correct, most customers rarely experience a defect.  Interestingly, when I think of the times where I did notice a defect in the Office products, it is when I use a tangential feature like mail merge in Word.

Summary

The 80-20 rule, when applied to testing, can have great effect on your quality and user experience.  In product delivery environments where there is constant pressure to deliver faster, applying the 80-20 rule helps to prioritize your efforts and makes it possible to deliver faster while satisfying a majority of your customers.  While our culture has embraced faster is always better, this is one tool to help identify your priorities with improved product success.

Watch the presentation.  It’s very good.

Similar to hardware fatigue, software exhibits fatigue under continuous usage as well.  For example, a memory leak in an application that runs for a very short time wouldn’t have any consequences.  However, after eight continuous hours of usage, a memory leak could render the application in operable.

Objectives of Soak Testing

Soak testing achieves a number of quality objectives by surfacing errors that would normally go undetected with simple usage.  By achieving these objectives, it is possible to release applications with unmatched quality in your market niche.

Memory

A memory leak of 20 bytes per minute isn’t easily detected after a few minutes.  It’s not easily detected after an hour with only a little over 1k of leakage, but after 24 hours, it’s becoming obvious that memory is allocated without being freed.  

For some applications a 24K memory leak over a 24 hour period may be okay.  However, it’s not just the heap that you may be concerned about.  In early versions of the Windows Operating System, the number of windows, brushes, and fonts that you could create was limited to 64K as the User and GDI data segments were limited to 64K each. 

If your application is leaking window handles and GDI handles, your application could exhaust the available memory to create a new window or font long before the heap is exhausted.  Not only does this adversely impact your application, it also adversely impacts any of the other applications running at the same time. 

As I recollect, exiting the application did not automatically free up these objects, so applications having these errors would eventually degrade performance of the operating system, and a reboot would be required to correct the condition.

While this is no longer a concern for the current Windows operating systems, it may be a problem for other third party libraries that you may be using in your application.

Race Conditions

Soak testing uncovers race conditions.  A race condition is an error in a calculation due to the sequence or timing of execution in multithreaded applications.  Continuous execution of an application raises the probability that the erroneous sequence is repeated many times.  Through repeated occurrences, it becomes possible for the software team to detect the condition.

Deadlocks

Soak testing uncovers deadlocks. In its simplest form, a deadlock is a symptom of a multithreaded application where two threads are waiting on each other to relinquish a resource before either one can continue to execute.  Deadlocks often require the right conditions to exist before they surface.   Through continuous execution over long periods of time, the probability of the deadlock condition being established increases.

Uninitialized Variables

Uninitialized variables are another source of unstable applications.  In particular, an uninitialized pointer can reveal itself in many ways.  If the uninitialized value points within the application data segment, using the pointer will corrupt the values of other global variables resulting in strange application behavior.

Sometimes this may manifest itself as a code exception when an unitialized pointer happens to point to the return address on the stack and the value is changed.  This happens to be a more difficult error to analyze.

Or in other instances, the value points outside the data segment and the application will terminate with a memory exception.  This form of a memory exception is actually the easiest manifestation of this problem to analyze and correct.

Performance Degradation

There are many manifestations of performance degradation.  Continuous execution flushes out many of them.  The most common sources of performance degradation are inefficient memory usage, throughput bottlenecks, and inefficient collection management.

• Inefficient Memory Usage – There are a number of ways this can manifest itself in an application: usually it involves a lot of unnecessary copying of data.  In some applications, the amount of memory copied to perform an operation may grow with repeated execution.
• Throughput bottlenecks - In real-time applications where data is constantly processed, data will back up when data arrives faster than it is processed.  If the application operates for only a short period of time, it can easily go unnoticed.
• Inefficient collection management - There’s nearly always some sort of data management in an application that makes the use of collections, whether it’s an array, a hash table, or linked list.  For some applications, these collections may build to appropriately large numbers of elements.  It’s then that you come to learn that the search algorithms or even the collection is the wrong solution for this problem.

Summary

Soak testing is the only reliable technique that I’m aware of for surfacing the class of application defects formerly described.  The objective of soak testing is not to assess the correctness of the application; it’s not to assess whether feature requirements have been correctly satisfied.  The objective of soak testing is surface defects that will impair application reliability.

Include soak testing as part of your test strategy along with quality release criteria to realize a dramatic improvement in the quality of your product releases.

We experienced some technical difficulties at one of our data centers which resulted in some users missing stats data. We are attempting to recover any lost data, and restore the service to full working capacity.

Thank you for bearing with us during this time.

Christopher C. Shannon

whos.amung.us

Christopher responded quite rapidly, and I appreciated his frank response.  However, I grew impatient waiting for a fix, and I’ve been thinking about upgrading to feedjit.com for a while now. I hesitated because I didn’t want to lose all the data that has accumulated since I’ve been using maps.amung.us.  While I was drawn to feedjit.com‘s widget because I like the map better, it wasn’t enough to make me switch even though maps.amung.us hadn’t made a single improvement since I began using the widget ten months ago.  However with this defect, there was nothing to lose by giving feedjit.com‘s widget a try, and so I did. 

The web changes everything.  Now that so much software is offered for free, the barrier for switching vendors based on price is gone.  Consequently, high quality is even more important when the barrier to switching vendors is nearly zero.  Quality and useful features on the first delivery are a competitive advantage, and they are often more important to consumers than new/enhanced features delivered frequently.  Incremental, frequent enhancements to satisfied customers is simply not required, but if you get your quality wrong or deliver the wrong features, you will lose customers — even formerly satisfied customers.

“I’d rather drive a Pinto that works every day than a Lexus that always breaks down,” said an unsatisfied customer of a company that delivered a product plagued with quality problems.  For him leading edge was less important than quality, or to put in other way, leading edge features that don’t work is like not having the features in the first place.  That’s true for most customers. 

It appears that the software development community believes that customers have a high tolerance for product defects.  How else can you explain the low quality that we often experience in the software products that we purchase?   I’ve often written about my experience with video editing software where it was difficult to find a product that could compile an edited sequence successfully.  Essentially, many of the video editing products at the time could not reliably complete the primary function for which people purchased them: compile sequences to a single AVI file.

Consumer Product Experience

What if you experienced similar quality in other products that you purchase?  What if you purchased a toaster that always burnt the toast?  What if you purchased a lawn mower that was difficult to start?  What if you purchased a car that frequently broke down?  What if you purchased new furniture with an obvious blemish?  Most of us would find this level of quality unacceptable unless, maybe they are steeply discounted.  But not in all these cases.  If you always get burnt toast, you’d want your money back even if the workaround is to use a timer and manually eject the toast when enough time has elapsed.  It sounds preposterous to even suggest that workaround is acceptable, but we often ask customers of our applications to accept similar preposterous workarounds for our software defects.

Now what would be your impression of the engineering team if in addition to all these defects, there are obvious misspellings and grammar errors in the on screen text, error messages, and product labels?  I bet your reaction would be similar to mine: of course the product doesn’t work, these people are incompetent.  They can’t even spell or write good English.

Workaround Defects

During the heat of a fast paced release, there is often pressure to defer fixing defects to future releases.  If the defect has some functional workaround, we often rightly lower the defect’s severity.  Defects with no workaround are obviously more severe than defects with a workaround.  However, this becomes dangerous when it’s argued that it is acceptable to release the product with broken features having a workaround. 

To bring some more clarity to the decision to release with or without the fix, it helps to draw some analogies to problems in other products that we use.  For example, if we have a feature in our software product that sporadically does not function and the workaround is to try it a second or third time to get it to work, would we find similar behavior in our new car to be acceptable?  If we purchase a new car and the radio would sporadically turn on or turn off when you press the power button and the workaround is to smack the radio hard once between the volume control and the tuner control, would you find that acceptable?  You’d want it fixed, and you’d be darn mad, right?  If we’d find the analogous workaround unacceptable in other products that we purchase, chances are that our customers will find it unacceptable in our software products too.

Cosmetic Defects

Another source of deferred defects is cosmetic defects.  Maybe a button isn’t in the right place, a message box has misspellings, or an edit control is too short.  Often these are rightly assigned low severity.  None of these prevents the functionality from working, but they make the product look bad.   If we release with any one of these defects, we likely would get away with it.  What if there were a thousand of these low severity defects?  Can you still safely release?

Let’s make the automobile analogy again.  Let’s assume you go to the dealer to pick up your new vehicle.  You walk around the car to inspect it making sure there aren’t any defects that would prevent you from driving the car off the premises.  As you walk around, you spot a minor surface scratch. Would you still drive off with the car?  Most people would, but I suppose a very small minority would not.  Now what if there were a large number of these surface scratches all over the vehicle and when you look at the vehicle from differing angles, the scratches noticeably detract from the new car finish and make it look unsightly.  Would you still drive the care off the premises? 

Leave a Good Impression

The release date nears, and you have to decide whether to slip the date or release as planned.  The team decides to release as planned with all the open cosmetic defects unaddressed and all the open feature defects with workaround solutions unaddressed.   If your customers rarely discover these problems, then you made the right decision to release.  On the other hand, if your customers are discovering problems frequently and becoming frustrated with using your product, then you made the wrong decision.  What impression do you think your customers will have of your team when in addition to all the defects preventing them from using your product successfully, the product is littered with many spelling and grammar errors?  Does the word professional come to mind?

Quality Matters

I read it in the blogs.  Many writing about software practices believe that quality is a trade off – especially if we are developing leading edge technologies, but really it isn’t.  Do you believe the astronauts of the space shuttle, a leading edge technology, have high expectations for quality?  How about the taxpayers of the space shuttle who are shouldering the funding for the program?  Do they find it acceptable to see their hard earned tax dollars destroyed in a disaster along with the precious lives of the shuttle astronauts?  While it’s difficult to prevent all disasters, we are more likely to have disasters when we have a casual attitude towards quality.

Even if your product reliably performs as advertised, do you still want to leave many cosmetic defects open in the release?  If there are enough cosmetic defects, it’s still going to leave your customers with a negative impression of your company and the team building the product.  What is the image of the company and the team that you want to communicate?  If the image you want to communicate is a company that has top talent and aims to deliver high quality and the best user experience, you better fix most of those cosmetic defects.  If you have lesser objectives, well, you can leave most of those cosmetic defects present.

Summary

As a consumer I value high quality.  When a company develops a reputation for high quality, I am more likely to purchase their products, and once I own their product, it’s difficult for me to switch to a competitor.  I can think of a number of once famous leading edge technology companies that failed to deliver leading edge quality.  Most of those companies that I’m thinking about right now are no longer in business, or they are struggling to survive. I was never a repeat customer of any of their products.  I often replaced their products quickly with a competitor’s.

When you are prepared to release with defects having a workaround, put yourself in the customers shoes; ask yourself, would I be satisfied with that workaround?  Would I be satisfied with an analogous workaround in any other consumer product?  When you are prepared to release with many cosmetic defects, ask yourself, what image would I have of the team and company if I were a user of this product?  How would I react with analogous cosmetic defects in any other consumer product?  Chances are your customers will react similar to you.  It’s best you fix them if you want to keep that customer

In software, where high quality continues to be uncommon, high quality is a competitive advantage.  Yes, delivering high quality takes time, but in my experience, low quality always takes longer and costs more.  As I see it, focusing on quality is a win-win strategy: it saves time and money, and it creates a positive reputation for your products and your company. Plus, your customers will be loyal for it.

Early in my career I had the good fortune to work under a management team that valued producing high quality applications – ones that rarely, if ever, crash under continual load.   The director of the department believed, not only in words but in action, that a defect discovered in the field is significantly more costly to fix than one discovered in development.  We lived by that rule, and we all believed in it.  

For this product, we did not have a sustaining engineering team to fix and deliver patches to the field because it was simply not needed. Over the course of the seven years of working on this team through multiple product deliveries, I recall only two or three issues from the field that required an engineering fix.   Now think of all the features you could deliver by investing the money saved in a sustaining engineering team into new feature development. That’s what we did.

I’d like to describe the six practices required to produce and release stable applications.  The level of stability that you achieve in your products will correlate to the degree to which you commit to these practices.  The strategy by itself isn’t enough to achieve great results.  Great results require discipline, persistence, and a strong will, and most importantly, it requires a leader and a team that values quality above all other criteria.

Quality is Everyone’s Responsibility

While the quality assurance team has quality in its name, quality is everyone’s responsibility.  On this team we didn’t have a quality engineering team.  Developers wrote and executed the performance tests, regression tests, integration tests, and acceptance test plans.  While there was no mistaking that quality was everyone’s responsibility on this team, it isn’t necessary to have the developers perform double duty as the quality team, but they do need to be involved in the test activities. 

Unit testing is something that must stay with the developers, and it should not be shortchanged to secure a delivery.  The developers should have responsibility in reviewing the test designs produced by the quality team, and they should be involved in designing and executing the performance tests.  If the need arises, developers should be available to pitch in to support the test execution effort.  Developers always should be available and open to questions from the testers.  If the engineering culture is that developers only write code, the team isn’t prepared to produce higher levels of quality.

Practice Defensive Programming

This approach was popular when I started working in the field, but it seems to have lost its luster.  I guess because it appears to take more time.  Defensive programming is about writing code to proactively detect error conditions and respond to them in sensible ways to prevent the program from failing or to have the program fail gracefully.  There are two primary benefits to defensive programming:  the developers become more focused on preventing errors, and when errors do surface, they can be solved quickly as the error is detected closer to where the error materialized.

Instrument the code

Have a debug version of your code with instrumentation to facilitate debugging when error conditions are detected.  This improves the turnaround time for fixing defects when they do occur.  One of my favorite practices is liberal use of the ASSERT Macro in C.  This practice was made popular in the 90′s by Steve Maguire in his book titled, “Writing Solid Code.” Use the ASSERT macro to validate parameters passed into procedures and functions, and use it to assert the conditions upon exiting a procedure or function.  If you are detecting error conditions at the earliest instance in which they materialize, you will dramatically improve your team’s ability to find the root cause quickly. 

ASSERTS differ from defensive programming in two ways: they are only present in debug versions of the application, and they do not take corrective action; instead, they break into the debugger when the error condition is detected.

Do not limit instrumentation to ASSERT macros only.  Some other good practices are to have execution tracing logic, initializing memory when allocated, and object validation logic to name a few others.

Soak Test the Application

Soak testing is the key practice.  Soak testing is the practice of exercising the application under automated user control.  The goal of the automated testing is to put the application under continuous user load to surface problems that would only surface after repeated usage.  It is best if the user input is randomly generated, but it isn’t required to have good results.  

If the application processes input other than user input, it is necessary to have simulators to generate that input and to pump that into the application continuously while the application is under automated script control.   At least every morning the applications should be surveyed for error conditions.   These are some of the candidate error conditions to monitor:

1. Memory Leaks – is application memory usage continually growing?
2. Secondary storage leaks – is the application consumption of secondary storage continually growing?
3. File handle leaks – are the open file handles continually growing?
4. Lock ups – is the application still responding to user input or does it appear to be locked/hung?
5. Crashes – has the application committed an operating system protection violation?

If any of these conditions are detected, they should be analyzed and fixed immediately. Keeping the application running continuously under automated script control has precedence over all other issues.  Once the issue is fixed, a new build is released to the lab to undergo soak testing. 

Performance Test to Failure

Many teams will execute performance tests to validate that the performance objectives of the system have been achieved and stop there, but it’s not enough to release a stable application.  There is really no telling how your customers will finally use your application.  Performance test the application beyond the performance requirements until it fails. You may be surprised to find that the application can exceed the performance objectives significantly by addressing some minor issues revealed when the application is stressed beyond the requirements.

The final objective is that when the user exceeds the performance specification, the system handles the condition gracefully. It should not crash, and it should inform the user that performance has degraded, and that calculations are unreliable while it is operating in this overload state.  When the load settles within the bounds of the specification, the application should recover and begin to perform normally.

Have Stability Release Criteria

The release decision has to be objective, has to be reflective of the test results, and has to support the goal of releasing a stable product.   Here are some recommended stability release criteria.

1. Identify the length of time for the application to run uninterrupted under soak testing.  I like to set the target at a multiple of the expected user usage.  For example, if I’m testing a professional trading application, the work week is 5 business days or one week.  At a minimum, the application should run uninterrupted for 5 straight days, but I would try to have that application successfully run for 30 days or one month under soak testing.  The release build should have at least achieved the minimum criteria.
2. Have no known resource leaks in the application at release time: dynamic memory, secondary storage, file handles, or any other limited resource used by the application.
3. Have no known crashes at release time.

Summary

It is my belief that releasing a high quality product is the easiest way to differentiate your product, and low or sub par quality is the surest way to lose a customer forever.  It’s difficult to unglue a costumer from a reliable product even if it does not have all the bells and whistles of competitive products, but of course, it must have the essential features to be desirable.  We only have to look at the American auto industry for a great example of how low quality can destroy a once great American industry.

I recall a number of years back when I began to develop an interest in video editing on the PC.  After evaluating a number of products, I settled on Adobe Premiere.  While the quality of Adobe Premier has improved over time, the initial version that I worked with could not complete the job that it was advertised to do, and it was true for all the competitive products that I evaluated at the time.  I only settled on Adobe Premiere because it came bundled with the Matrox video capture card that I purchased: RT X10. 

My video editing needs aren’t demanding.  I didn’t need many bells and whistles.  All I needed was the ability to capture, to splice multiple clips and stills, to overlay audio, to edit some titles, and to insert transitions.  As an amateur editor, it couldn’t reliably satisfy my limited demands.  If I could have found a less feature laden product with higher quality at a premium price, I would have purchased it without hesitation.  I have no doubt if there was a competitor offering a competitive product with a reputation for high quality, the product would have been able to acquire significant market share.

Quality is a differentiator.  Grab the low hanging fruit; put these practices to use in your own projects, build stable applications, and you will differentiate your products from the competition.

Did you ever read one of those articles where the writer had instant credibility?  I recently stumbled upon one when I was looking to get my web site indexed on dmoz.org.  The article is titled “Nine Steps to Delivering Defect-Free Software” by Terence M. Colligan.  When I read the article I knew he isn’t just writing about somebody else’s experience because it mirrored my own experiences when I was writing software myself.  In the article he identifies nine traits contributing to defect free code.

1. “Believe Defect-Free Software is Possible.” It seems as though some people in the business have given up in delivering high quality.  They take it as a given that high quality is not possible.  I can’t tell you how often I’ve seen products shipped on a date.  I’ve read one blogger on this topic advocate that quality is a trade-off relaxed to achieve other priorities.  I don’t agree.

2. “Think Defect-Free Software is Important.” If you want to deliver more features to market quicker, start delivering higher quality.  Low quality is a drag on productivity.

3. “Commit to Delivering Defect-Free Software.”  To commit to delivering defect-free software you need quality release criteria.  I plan to explore this deeper in a future article.

4. “Design Your Code for Simplicity and Reliability.”  Yes, big up front design pays big dividends.  If your desire is a high quality product, you cannot skip this practice.

5. “Trace Every Line of Code When Written.”  It sounds tedious and laborious, and it is, but that’s life.  Do this if you desire high quality.

6. “Review Code by Programmer Peers.”  This will find a great deal of design and implementation flaws.   When done well, peer reviews are extremely effective, but it is often difficult for peers to invest enough time in the review to reap the dividends.

7. “Build Automated QA into Your Code.”  This will add up front overhead to your project, but it will ultimately save time.

8. “Build and Test Daily.”  I hit upon this some in my article “No Pain, No Gain.” This is a great practice.

9. “Use Automated Checking Wherever Possible.” It’s also a good idea to set some standards for what compiler or lint warnings must be fixed before releasing software to the build.

I’ve experienced all of these practices to varying degrees, and they are all indispensable in attaining the goal of delivering defect free software.  Most importantly, you have to believe defect-free code is possible and you have to want to achieve it to be successful.  Those are the two most important characteristics for defect-free success.  Give the article a read and start believing and wanting.  Defect-free software is possible if only more software managers would believe in the ideal and want to achieve it.  Low quality software is always a management problem.

Reference

Nine Steps to Delivering Defect-Free Software

Have you ever been so frustrated with a piece of software that you wanted to scream?  I’m sure it’s happened to you.  You’ve just completed a part of your project that you’ve worked so hard to get right and then – poof — magically the application goes away and you just lost all of your important changes. It happened to me when I started to get involved in video editing. It was difficult to find an application that I could successfully complete an entire project.  I evaluated many of them before I settled on Adobe Premiere, and even that was buggy; it happened to be the best of the field of video editing applications.

I recall editing a sequence in Adobe Premiere consisting of a collection of photos overlaid with music.  When I compiled the sequence into an AVI file, the video would playback with a black screen: all the photo sequences were gone from the AVI.  It seemed to be random, I could get some projects to work and others not. After analyzing the problem, it became apparent that the program had a memory management problem as memory would continue to grow as it compiled the sequence; short sequences would compile successfully, long sequences would fail because memory would be exhausted.  I found it frustrating that the company could release the software with such an obvious flaw in a feature that I needed.

My success with delivering software to high quality standards has been quite different than my experiences with many PC applications.  One of the teams that I worked on was particularly noteworthy for the level of  quality that we delivered.  This application was a Windows based application, having its origins in Windows 1.0.  It was a real-time financial application that delivered financial news and trading information 24 x 7 to the global financial community.  At it’s zenith the installed base was approximately 250,000 users.

The processing demands of the application were constant as the application processed a continuous stream of financial data as well as a respond to user requests.  In the 8 or so years that I worked on this product, as both a developer and manager, I recall less than 5 defects escalating to engineering and none of them were stability issues.  We didn’t need nor staff a sustaining engineering team.  The application just worked! 

While there were many practices that contributed to this high level of quality, the practices I’m about to describe were essential in achieving these results.  Basically, they are the practices that if skipped, high levels of quality would not be achievable to the degree that we achieved them.  I’ve had the opportunity to repeat these practices at other companies and on other assignments, and the results are always the same –exceptional quality!

Foster and support a high commitment to quality 

Quality was without a doubt the number one release criteria for this product, and everyone on the team knew it.   If we found a critical defect in the days before the release, we would slip the release if it could not be fixed prior to ship date. 

No developer wanted to be tagged with the defect that held up the release.  Consequently, the developers took unit testing code very seriously.  We were extremely thorough in unit testing and defect fixing.

The benefit of having high quality standards is that the team members motivate themselves to perform at high levels.  Without a high level of management commitment to quality, high levels of quality are just not possible.  There is no compromising on this requirement to achieve high levels of quality.  I’ve never seen a product release with high quality where the management did not demonstrate a firm commitment to quality.

Release a build to test early and continue to release builds to test weekly at a minimum, daily if possible

One of the key milestones in our schedules was the delivery of that first build to test.  The more time the application spends in test, the more opportunity you have to find defects.  The director of the product had a plaque on his wall that read, “Good Software Takes Time.”  It takes time to find and fix all the defects required to ship a high quality product.  Start finding defects as early as possible and continue to find and fix new defects as you add code to each new build.

Have a Soak testing strategy

Once that first build was ready for test, it would be bombarded 24 x 7 with automated test scripts.   The test scripts were updated with every release, and our objective was both depth and breadth of coverage to the extent possible with the tools we were using at the time.  There were two goals in soak testing: one to flush out all stability issues, and two, to release a build that would withstand the rigors of our automated tests indefinitely.

Every morning we would inspect all the machines for tell tale signs of memory leaks, crashes, deadlocks, and performance degradation.    If issues were found, we’d record the details, raise defects and even begin fixing them depending on the criticality.

Develop highly skilled debuggers

One of the requirements for reaching the senior programmer ranks was to demonstrate top notch debugging skills.   To demonstrate top notch debugging skills you had to demonstrate that you could debug a stack dump.  This gets you out of the need to repeat issues in order to debug them.  Consequently, the turnaround time for fixing critical issues normally required a day or less to solve rather than the days and weeks required when having to repeat the fault to find the root cause.  While this doesn’t totally remove the team from having to find the repeatable sequence of events to solve a defect, it does eliminate over 90% of them.

Leave no bug behind

Our philosophy here was if we saw the defect once, our customers would see it many times.  We were tenacious at finding hard to replicate defects.   We would literally spend days finding the repeatable sequence for defects that surfaced infrequently.  We considered it a failure if we were unsuccessful in finding and fixing a defect, so we rarely had a defect that we could not find and fix.   I’m not suggesting you need to fix every defect you find before you release; just don’t give up on the high severity ones because they are difficult to find or fix.

Find the applications breaking points

One objective of testing should be to find where your application breaks because if you don’t, your customers will.   Not only did our testing look to verify that the functionality was delivered to specification, every test looked to find the breaking points in the application.  If functionality allowed for the creation of things, we attempted to find how many we could create before it would break.  If we were performance testing, we would continue to stress the application until it no longer could sustain the input – even when the application long passed the performance requirements.  This uncovered all sorts of weaknesses that would have been found in even less demanding usage, but doing so flushed them out quickly.

Have established release criteria

We had established release criteria based on quality metrics.  This prevented the team from releasing a product that was not ready due to the pressures of hitting a release date.   We were actually good about making our dates; we typically released within a month of our planned ship dates for release cycles that spanned from 6 to 9 months.

Conclusion

Delivering a high quality application takes a serious commitment to quality and a lot of hard work and sweat.  In sports they often say, “no pain, no gain.”   That idea is similarly true for delivering high quality software.  There are no short cuts to high quality.  If you’re unwilling to deal with the pain of slipping a release to deliver to high quality standards or the pain of analyzing and solving a difficult defect, you will be unsuccessful in delivering a high quality product.  For those willing to make the commitment, they will be rewarded with a successful product, highly satisfied customers, low support costs, and high employee moral.  Aren’t those reasons enough to make the commitment?

• How to Break Web Software: Functional and Security Testing of Web Applications and Web Services. Book & CD
• Metrics and Models in Software Quality Engineering (2nd Edition)
• Peer Reviews in Software: A Practical Guide
• Software Quality Assurance: Principles And Practice
• Design for Reliability (Electronics Handbook Series)
• Solid Software
• High Quality Low Cost Software Inspections
• Debugging: The Nine Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems
• Software Verification and Validation: A Practitioner’s Guide (Artech Computer Science Library)