Have you ever been so frustrated with a piece of software that you wanted to scream?  I’m sure it’s happened to you.  You’ve just completed a part of your project that you’ve worked so hard to get right and then – poof — magically the application goes away and you just lost all of your important changes. It happened to me when I started to get involved in video editing. It was difficult to find an application that I could successfully complete an entire project.  I evaluated many of them before I settled on Adobe Premiere, and even that was buggy; it happened to be the best of the field of video editing applications.

I recall editing a sequence in Adobe Premiere consisting of a collection of photos overlaid with music.  When I compiled the sequence into an AVI file, the video would playback with a black screen: all the photo sequences were gone from the AVI.  It seemed to be random, I could get some projects to work and others not. After analyzing the problem, it became apparent that the program had a memory management problem as memory would continue to grow as it compiled the sequence; short sequences would compile successfully, long sequences would fail because memory would be exhausted.  I found it frustrating that the company could release the software with such an obvious flaw in a feature that I needed.

My success with delivering software to high quality standards has been quite different than my experiences with many PC applications.  One of the teams that I worked on was particularly noteworthy for the level of  quality that we delivered.  This application was a Windows based application, having its origins in Windows 1.0.  It was a real-time financial application that delivered financial news and trading information 24 x 7 to the global financial community.  At it’s zenith the installed base was approximately 250,000 users.

The processing demands of the application were constant as the application processed a continuous stream of financial data as well as a respond to user requests.  In the 8 or so years that I worked on this product, as both a developer and manager, I recall less than 5 defects escalating to engineering and none of them were stability issues.  We didn’t need nor staff a sustaining engineering team.  The application just worked! 

While there were many practices that contributed to this high level of quality, the practices I’m about to describe were essential in achieving these results.  Basically, they are the practices that if skipped, high levels of quality would not be achievable to the degree that we achieved them.  I’ve had the opportunity to repeat these practices at other companies and on other assignments, and the results are always the same –exceptional quality!

Foster and support a high commitment to quality 

Quality was without a doubt the number one release criteria for this product, and everyone on the team knew it.   If we found a critical defect in the days before the release, we would slip the release if it could not be fixed prior to ship date. 

No developer wanted to be tagged with the defect that held up the release.  Consequently, the developers took unit testing code very seriously.  We were extremely thorough in unit testing and defect fixing.

The benefit of having high quality standards is that the team members motivate themselves to perform at high levels.  Without a high level of management commitment to quality, high levels of quality are just not possible.  There is no compromising on this requirement to achieve high levels of quality.  I’ve never seen a product release with high quality where the management did not demonstrate a firm commitment to quality.

Release a build to test early and continue to release builds to test weekly at a minimum, daily if possible

One of the key milestones in our schedules was the delivery of that first build to test.  The more time the application spends in test, the more opportunity you have to find defects.  The director of the product had a plaque on his wall that read, “Good Software Takes Time.”  It takes time to find and fix all the defects required to ship a high quality product.  Start finding defects as early as possible and continue to find and fix new defects as you add code to each new build.

Have a Soak testing strategy

Once that first build was ready for test, it would be bombarded 24 x 7 with automated test scripts.   The test scripts were updated with every release, and our objective was both depth and breadth of coverage to the extent possible with the tools we were using at the time.  There were two goals in soak testing: one to flush out all stability issues, and two, to release a build that would withstand the rigors of our automated tests indefinitely.

Every morning we would inspect all the machines for tell tale signs of memory leaks, crashes, deadlocks, and performance degradation.    If issues were found, we’d record the details, raise defects and even begin fixing them depending on the criticality.

Develop highly skilled debuggers

One of the requirements for reaching the senior programmer ranks was to demonstrate top notch debugging skills.   To demonstrate top notch debugging skills you had to demonstrate that you could debug a stack dump.  This gets you out of the need to repeat issues in order to debug them.  Consequently, the turnaround time for fixing critical issues normally required a day or less to solve rather than the days and weeks required when having to repeat the fault to find the root cause.  While this doesn’t totally remove the team from having to find the repeatable sequence of events to solve a defect, it does eliminate over 90% of them.

Leave no bug behind

Our philosophy here was if we saw the defect once, our customers would see it many times.  We were tenacious at finding hard to replicate defects.   We would literally spend days finding the repeatable sequence for defects that surfaced infrequently.  We considered it a failure if we were unsuccessful in finding and fixing a defect, so we rarely had a defect that we could not find and fix.   I’m not suggesting you need to fix every defect you find before you release; just don’t give up on the high severity ones because they are difficult to find or fix.

Find the applications breaking points

One objective of testing should be to find where your application breaks because if you don’t, your customers will.   Not only did our testing look to verify that the functionality was delivered to specification, every test looked to find the breaking points in the application.  If functionality allowed for the creation of things, we attempted to find how many we could create before it would break.  If we were performance testing, we would continue to stress the application until it no longer could sustain the input – even when the application long passed the performance requirements.  This uncovered all sorts of weaknesses that would have been found in even less demanding usage, but doing so flushed them out quickly.

Have established release criteria

We had established release criteria based on quality metrics.  This prevented the team from releasing a product that was not ready due to the pressures of hitting a release date.   We were actually good about making our dates; we typically released within a month of our planned ship dates for release cycles that spanned from 6 to 9 months.

Conclusion

Delivering a high quality application takes a serious commitment to quality and a lot of hard work and sweat.  In sports they often say, “no pain, no gain.”   That idea is similarly true for delivering high quality software.  There are no short cuts to high quality.  If you’re unwilling to deal with the pain of slipping a release to deliver to high quality standards or the pain of analyzing and solving a difficult defect, you will be unsuccessful in delivering a high quality product.  For those willing to make the commitment, they will be rewarded with a successful product, highly satisfied customers, low support costs, and high employee moral.  Aren’t those reasons enough to make the commitment?

• How to Break Web Software: Functional and Security Testing of Web Applications and Web Services. Book & CD
• Metrics and Models in Software Quality Engineering (2nd Edition)
• Peer Reviews in Software: A Practical Guide
• Software Quality Assurance: Principles And Practice
• Design for Reliability (Electronics Handbook Series)
• Solid Software
• High Quality Low Cost Software Inspections
• Debugging: The Nine Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems
• Software Verification and Validation: A Practitioner’s Guide (Artech Computer Science Library)