He was one of the top software engineers of a talented team.  His assignment was to improve the throughput of our real-time adaptive performance monitoring algorithms.  It wasn’t going to be easy as the application was already processing messages at a high rate.

The application had processing throughput of approximately 1,000 400-byte messages per second. Our goal was to double it to approximately 2,000 messages per second.  When he completed the assignment, it was processing over 3,000 messages a second.

Approach

How did he do it?  Well…it didn’t start so smooth.  He dug right in, and that was the problem.  He knew the code base well, and he believed with a hunch, some experimentation, some feedback, and some refactoring that he could improve the throughput and reach the goal.

There was only one thing: it wasn’t working. After a few days, I asked him about his approach and where he believed the bottleneck to be.  He said that he tried a couple of ideas, but he didn’t get the improvements that he expected, and he had a few additional ideas to try.

I asked, “Ravi, why don’t you profile the code?  While I have no doubt that these areas need improving, you might be optimizing code that’s consuming only 5% of the overall processing, and optimizing those areas are not going to lead to the required solution.”  He understood, but he felt confident in his understanding of the code and his approach.

I allowed him to continue.  Ravi was a top gun.  Very few engineers could debug a UNIX stack dump and reverse engineer the Solaris OS like Ravi could.   His designs were solid, and his deliveries were always high quality.   He earned the opportunity to continue with his approach even though I wasn’t supportive of it, but there were limits.

Two weeks later, he was still working on it, and the performance improvement was only marginal.  I said, “Ravi, it’s time to profile the code.”  Reluctantly, he agreed.

The Application

Some technical details of the application would be helpful at this point.

The application was a network performance monitoring application. It performed passive surveillance of the edge network nodes.  It was a multi-tiered, distributed application using CORBA as the middleware technology.  The language was C++, and it was deployed on Sun/Solaris equipment.

Upfront Analysis

It took about a day to prepare the environment and the application for performance monitoring.  After a couple of runs, it was obvious what processing required optimization.

Once the message was received from the edge devices at the data receiver, the application parsed and transformed the network message into an application message object and forwarded it to the upstream processes.  The performance bottleneck was with the marshalling of this object.  As the message object moved from each distributed process, this expensive marshalling was repeated multiple times, and it consumed a significant percentage of the overall processing of the message.

Solution

The solution was quite simple: reformat the message into a byte array: essentially, “char Message[500];” was the only member of the message object.   Prior to the change, each field in the message had its own member variable.  The network message was reformatted  into the byte array to avoid the overhead of re-parsing the message as it made its way through the processing path, which was the original intent of the message object.

At the beginning of the byte array a field identifier, offset pair was inserted for each field in the message with the offset being an index from the beginning of the byte array to the location of the field data.  With a lookup and a simple cast, the data was easily extracted.

The overhead for marshalling the byte array was significantly less with CORBA.   With this approach, the performance more than tripled to processing over 3000 messages per second.

Finally, the duration required to profile, to analyze, to design, to code and to release the change into the next build took about 3 to 5 days. Had Ravi started with upfront analysis, he would have saved 2 weeks of effort, and upfront analysis arrived at a solution faster than the approach with no formal analysis.  None of the prior work provided any insight to this bottleneck and solution.  In fact, it was a surprise.

Analysis is Hard

Some of you may be saying well, this is quite obvious, but is it?  I’ve had similar scenarios repeated with other developers throughout the years.  Developers want to write code, but writing code is really the easy part of the programmer’s job. 

The hard part is creating the model for the solution, identifying the objects, and their methods.  It’s the analysis and the design that’s the difficult part of software development. 

Sure, not all problems are difficult to model, but the modeling is more difficult than the coding.  Any programmer can transform a good design into working code, but any programmer cannot transform a problem/requirement into a good model.

For example, making the code changes to fix a defect is the easy part.  It takes days – sometimes even months – of analysis to understand the root cause of a difficult defect.  Once the root cause is identified, the solution often only requires changes to a couple of lines of code or sometimes to add, subtract, or change a single character. 

Writing code is easy.  It’s the understanding that’s the hard part;  it’s the transformation of a problem into a model that’s the difficult part. The code is simply the realization of that model and understanding.

Simplicity

One of the twelve Principles behind the Agile Manifesto is “Simplicity – the art of maximizing the amount of work not done – is essential.”  Many interpret this to mean skipping documentation and/or skipping upfront analysis and design – essentially skipping any manifestations of formalism.  

“The art of maximizing the amount of work not done” is best achieved by working smarter.  Achieving simplicity requires an investment in upfront analysis, and like all investments, they are rewarding when invested well.  

Having poor results with upfront analysis is not anymore evidence that upfront analysis should skipped than having poor code is evidence that coding should be skipped.  Analysis is essential to writing code.  Poor results with upfront analysis is simply evidence that you need to do analysis better.

Summary

Work smarter, not harder is common wisdom for solving difficult problems.  Analysis is the process of intelligently applying our knowledge to solve difficult problems. Upfront analysis saves time.  It’s about working smarter, not harder.

In the case of the opening example, the savings from upfront analysis is 66% of the duration and effort.  To look at it another way, the required duration and effort without upfront analysis is indeterminate. For all I know, if Ravi had continued with his original approach, he may have continued to work on it for another two weeks or even more.

Skip upfront analysis if you must, but as Glen B. Alleman, author of Herding Cats, is fond of saying, “Don’t do stupid things on purpose.“