Soak testing is the practice of exercising an application continuously under automated test with the primary objective of uncovering defects.  In my experience, it’s the best practice for delivering reliable applications: applications that never seem to crash or exit abruptly.

Similar to hardware fatigue, software exhibits fatigue under continuous usage as well.  For example, a memory leak in an application that runs for a very short time wouldn’t have any consequences.  However, after eight continuous hours of usage, a memory leak could render the application in operable.

Objectives of Soak Testing

Soak testing achieves a number of quality objectives by surfacing errors that would normally go undetected with simple usage.  By achieving these objectives, it is possible to release applications with unmatched quality in your market niche.

Memory

A memory leak of 20 bytes per minute isn’t easily detected after a few minutes.  It’s not easily detected after an hour with only a little over 1k of leakage, but after 24 hours, it’s becoming obvious that memory is allocated without being freed.  

For some applications a 24K memory leak over a 24 hour period may be okay.  However, it’s not just the heap that you may be concerned about.  In early versions of the Windows Operating System, the number of windows, brushes, and fonts that you could create was limited to 64K as the User and GDI data segments were limited to 64K each. 

If your application is leaking window handles and GDI handles, your application could exhaust the available memory to create a new window or font long before the heap is exhausted.  Not only does this adversely impact your application, it also adversely impacts any of the other applications running at the same time. 

As I recollect, exiting the application did not automatically free up these objects, so applications having these errors would eventually degrade performance of the operating system, and a reboot would be required to correct the condition.

While this is no longer a concern for the current Windows operating systems, it may be a problem for other third party libraries that you may be using in your application.

Race Conditions

Soak testing uncovers race conditions.  A race condition is an error in a calculation due to the sequence or timing of execution in multithreaded applications.  Continuous execution of an application raises the probability that the erroneous sequence is repeated many times.  Through repeated occurrences, it becomes possible for the software team to detect the condition.

Deadlocks

Soak testing uncovers deadlocks. In its simplest form, a deadlock is a symptom of a multithreaded application where two threads are waiting on each other to relinquish a resource before either one can continue to execute.  Deadlocks often require the right conditions to exist before they surface.   Through continuous execution over long periods of time, the probability of the deadlock condition being established increases.

Uninitialized Variables

Uninitialized variables are another source of unstable applications.  In particular, an uninitialized pointer can reveal itself in many ways.  If the uninitialized value points within the application data segment, using the pointer will corrupt the values of other global variables resulting in strange application behavior.

Sometimes this may manifest itself as a code exception when an unitialized pointer happens to point to the return address on the stack and the value is changed.  This happens to be a more difficult error to analyze.

Or in other instances, the value points outside the data segment and the application will terminate with a memory exception.  This form of a memory exception is actually the easiest manifestation of this problem to analyze and correct.

Performance Degradation

There are many manifestations of performance degradation.  Continuous execution flushes out many of them.  The most common sources of performance degradation are inefficient memory usage, throughput bottlenecks, and inefficient collection management.

• Inefficient Memory Usage - There are a number of ways this can manifest itself in an application: usually it involves a lot of unnecessary copying of data.  In some applications, the amount of memory copied to perform an operation may grow with repeated execution.
• Throughput bottlenecks - In real-time applications where data is constantly processed, data will back up when data arrives faster than it is processed.  If the application operates for only a short period of time, it can easily go unnoticed.
• Inefficient collection management - There’s nearly always some sort of data management in an application that makes the use of collections, whether it’s an array, a hash table, or linked list.  For some applications, these collections may build to appropriately large numbers of elements.  It’s then that you come to learn that the search algorithms or even the collection is the wrong solution for this problem.

Summary

Soak testing is the only reliable technique that I’m aware of for surfacing the class of application defects formerly described.  The objective of soak testing is not to assess the correctness of the application; it’s not to assess whether feature requirements have been correctly satisfied.  The objective of soak testing is surface defects that will impair application reliability.

Include soak testing as part of your test strategy along with quality release criteria to realize a dramatic improvement in the quality of your product releases.