As we saw in the last essay, the 80:20 rule seemingly appears in many circumstances.   When I was attending college and working as a programmer during the 80′s, there were some commonly accepted tenets that guided our software development processes and behaviors. 

• 80% of development effort for a product is spent in maintenance, while only 20% is devoted towards delivering the first release.
• It is many times more costly to remedy a defect in the later phases of the SDLC than in the earlier phases.

We didn’t question these beliefs much because they were commonly accepted as true.  Our own experiences supported these beliefs as well.  I’ve spent 80+% of my career maintaining software while the amount of time devoted to delivering a first release has been small.  I bet most of you reading this can say the same thing.  Entirely new product development, where you start from a blank slate, is rare in this business.

80% Maintenance

Think about it: how often do you purchase an entirely new application?  Most of my software purchases are for the newest release of a product that I already own.  Maybe I’m a Troglodyte, but I haven’t purchased an entirely  new application  in many years. I have, though, upgraded to the latest version of many applications within the last year.

In the enterprise space, many projects are about upgrading to the latest version of the Windows OS, Oracle Database, Apache web server, or ERP application, to name a few.  Even in the software product development space, less effort is devoted to rolling out entirely new tools and libraries than effort devoted to upgrading to the latest versions.

Some may say yes, but being first in a space has many advantages.  Does it?  Who remembers VisiCalc?  Turbo Pascal?  Netware?  Netscape?  CPM? There are litanies of firsts that are distant memories.  There’s an advantage to being second in a space.  Google in Search, Microsoft in browsers and office applications, and Dell in Personal Computers are a few examples where being 2^nd, 3^rd, or later to market was not a disadvantage in their eventual success and market domination.

Realizing that 80+% of development effort is devoted to maintenance, the old school developers organized their processes and priorities accordingly.  Being first is rarely the most important objective.  While being first has it’s virtues, dominating your market is a higher priority, and too often the compromises made to succeed at being  first compromises your ability to dominate the market.

The Old School Manifesto

Here’s what the old school programmers and leaders emphasize in their application development efforts:

• Creating a lasting architecture over quick construction of the first build. When done successfully, this saves time, improves quality, and improves time to market not only for the first release but for each successive release.
• Understanding the customer value over doing what the customer tells you. Very often, real customers don’t know exactly what they want nor do they know what’s possible. Customers are sometimes too busy to tell you what it is that they want. It is the development team’s responsibility to understand the need and to use their knowledge and intelligence to create a superior solution. If you’re expecting your customers to tell you everything, you aren’t the team to be building the next thing.
• Emphasizing the up front work over constructing something quickly. Devote 60% of your effort to the upfront work, requirements and design; because when you do it successfully, the last 40% is a no brainer.
• Thinking before acting over acting before thinking. God gave us intelligence for a purpose. Understanding, prioritizing, strategizing, and planning support success.
• Analysis over just-jump-right-in-and-do-something. It always saves time, money, and heartache.
• Believing that if an error happens in the lab once your customers will see it many times, over rebooting the machine and hoping it never happens again. I know; many defects are difficult to analyze and fix. I’ve worked through many nights addressing difficult defects. Just do it. Eventually difficult defects become easy to analyze and fix, but first you must invest to get there.
• Doing superior work over doing just enough. Superior work inspires good people, and motivates not so good people to perform better. I know just enough is fashionable today, but fashionable is just a passing trend.
• Quality over hitting a date. If quality made Japan the top auto manufactures of the world while being 2^nd to market, it must be telling us something. Listen carefully!
• The future over the present. Developing a product is an investment in the future. Investments are rewarding when good decisions about the future are made in the present.
• Change management over change anytime. Change is a fact of life. We get old; we move; things change. Making good decisions about change is necessary for repeatable success. Most changes are not reactions to changing markets but reactions to the leadership’s change in thinking. This too often reflects inadequate analysis.
• Teamwork over individuals. Individuals come and go in an organization but teams remain. Teamwork is the hallmark of a successful product team. Process is a manifestation of good teamwork. Good teamwork is greater than the sum of its parts. It’s through teamwork that superior products are made and delivered. Value the team and the process that supports teamwork.
• Value over commitments. The ultimate goal of every project is to deliver value. Meet your commitments without delivering value, and all you’ll have is a failed project.

Think About It

It’s become fashionable today to live in the present: just look at the debt our society carries from corporations, to individuals, and to governments.   All are carrying unprecedented amounts of debt.  We act and live today like there will be no tomorrow or that tomorrow doesn’t matter.   Would the worldwide financial crisis have materialized if corporate leaders concerned themselves with tomorrow?  When you concern yourself with tomorrow, the present is normally better as tomorrow unfolds into the present.  At the root of old school values was the idea that we are all working to build a better tomorrow?  It’s time that our culture reaffirms that commitment.  Don’t you think?

What would you add or change in the old school manifesto ?

1. If you are following one of the Agile methodologies, are you satisified with your results?  If not, where is it failing you?
2. What do you find to be the best aspect of the methodology that you follow?
3. If you could change one thing about the way your team practices software development, what would you change?

The application had processing throughput of approximately 1,000 400-byte messages per second. Our goal was to double it to approximately 2,000 messages per second.  When he completed the assignment, it was processing over 3,000 messages a second.

Approach

How did he do it?  Well…it didn’t start so smooth.  He dug right in, and that was the problem.  He knew the code base well, and he believed with a hunch, some experimentation, some feedback, and some refactoring that he could improve the throughput and reach the goal.

There was only one thing: it wasn’t working. After a few days, I asked him about his approach and where he believed the bottleneck to be.  He said that he tried a couple of ideas, but he didn’t get the improvements that he expected, and he had a few additional ideas to try.

I asked, “Ravi, why don’t you profile the code?  While I have no doubt that these areas need improving, you might be optimizing code that’s consuming only 5% of the overall processing, and optimizing those areas are not going to lead to the required solution.”  He understood, but he felt confident in his understanding of the code and his approach.

I allowed him to continue.  Ravi was a top gun.  Very few engineers could debug a UNIX stack dump and reverse engineer the Solaris OS like Ravi could.   His designs were solid, and his deliveries were always high quality.   He earned the opportunity to continue with his approach even though I wasn’t supportive of it, but there were limits.

Two weeks later, he was still working on it, and the performance improvement was only marginal.  I said, “Ravi, it’s time to profile the code.”  Reluctantly, he agreed.

The Application

Some technical details of the application would be helpful at this point.

The application was a network performance monitoring application. It performed passive surveillance of the edge network nodes.  It was a multi-tiered, distributed application using CORBA as the middleware technology.  The language was C++, and it was deployed on Sun/Solaris equipment.

Upfront Analysis

It took about a day to prepare the environment and the application for performance monitoring.  After a couple of runs, it was obvious what processing required optimization.

Once the message was received from the edge devices at the data receiver, the application parsed and transformed the network message into an application message object and forwarded it to the upstream processes.  The performance bottleneck was with the marshalling of this object.  As the message object moved from each distributed process, this expensive marshalling was repeated multiple times, and it consumed a significant percentage of the overall processing of the message.

Solution

The solution was quite simple: reformat the message into a byte array: essentially, “char Message[500];” was the only member of the message object.   Prior to the change, each field in the message had its own member variable.  The network message was reformatted  into the byte array to avoid the overhead of re-parsing the message as it made its way through the processing path, which was the original intent of the message object.

At the beginning of the byte array a field identifier, offset pair was inserted for each field in the message with the offset being an index from the beginning of the byte array to the location of the field data.  With a lookup and a simple cast, the data was easily extracted.

The overhead for marshalling the byte array was significantly less with CORBA.   With this approach, the performance more than tripled to processing over 3000 messages per second.

Finally, the duration required to profile, to analyze, to design, to code and to release the change into the next build took about 3 to 5 days. Had Ravi started with upfront analysis, he would have saved 2 weeks of effort, and upfront analysis arrived at a solution faster than the approach with no formal analysis.  None of the prior work provided any insight to this bottleneck and solution.  In fact, it was a surprise.

Analysis is Hard

Some of you may be saying well, this is quite obvious, but is it?  I’ve had similar scenarios repeated with other developers throughout the years.  Developers want to write code, but writing code is really the easy part of the programmer’s job. 

The hard part is creating the model for the solution, identifying the objects, and their methods.  It’s the analysis and the design that’s the difficult part of software development. 

Sure, not all problems are difficult to model, but the modeling is more difficult than the coding.  Any programmer can transform a good design into working code, but any programmer cannot transform a problem/requirement into a good model.

For example, making the code changes to fix a defect is the easy part.  It takes days – sometimes even months – of analysis to understand the root cause of a difficult defect.  Once the root cause is identified, the solution often only requires changes to a couple of lines of code or sometimes to add, subtract, or change a single character. 

Writing code is easy.  It’s the understanding that’s the hard part;  it’s the transformation of a problem into a model that’s the difficult part. The code is simply the realization of that model and understanding.

Simplicity

One of the twelve Principles behind the Agile Manifesto is “Simplicity – the art of maximizing the amount of work not done – is essential.”  Many interpret this to mean skipping documentation and/or skipping upfront analysis and design – essentially skipping any manifestations of formalism.  

“The art of maximizing the amount of work not done” is best achieved by working smarter.  Achieving simplicity requires an investment in upfront analysis, and like all investments, they are rewarding when invested well.  

Having poor results with upfront analysis is not anymore evidence that upfront analysis should skipped than having poor code is evidence that coding should be skipped.  Analysis is essential to writing code.  Poor results with upfront analysis is simply evidence that you need to do analysis better.

Summary

Work smarter, not harder is common wisdom for solving difficult problems.  Analysis is the process of intelligently applying our knowledge to solve difficult problems. Upfront analysis saves time.  It’s about working smarter, not harder.

In the case of the opening example, the savings from upfront analysis is 66% of the duration and effort.  To look at it another way, the required duration and effort without upfront analysis is indeterminate. For all I know, if Ravi had continued with his original approach, he may have continued to work on it for another two weeks or even more.

Skip upfront analysis if you must, but as Glen B. Alleman, author of Herding Cats, is fond of saying, “Don’t do stupid things on purpose.“

It’s suggested that the web has changed everything.  Whatever requirements there were for delivering desktop applications, the requirements for delivering web applications has changed.

For the web, the thinking goes; delivery of new features to customers is paramount to remain competitive.   Maybe it’s true, but what’s the evidence?  My own experience with popular web sites does not support this conclusion.

Online Examples

I’ve been using Citibank online for over 15 years now, and I use it the same as I did when I first began banking over the web.  The banking features haven’t changed for a long time until recently.  While some features of the application could have been vastly improved, I never felt the urge to change my bank because of the web application. 

But even if Citibank’s competitors offered a vastly improved online experience, I’m unlikely to change my bank because of the web application.  The Citibank online experience satisfies my need already, though admittedly improvements could be made.  Plus, how much better can you improve the features to pay bills, transfer money, and view transactions?

I’ve been using Ebay as a seller and a sometimes buyer for 10 years; the features that I use haven’t changed much nor do they need to.  Ebay has the one most important feature that cannot be realized in software improvements: access to a large community of consumers. 

I use Amazon.com frequently, and the content is way more valuable to me than the functionality delivered and so is the price of the products offered.  The feature of purchasing products on Amazon.com hasn’t changed much, nor does it need to.  I go to Amazon.com not because of the software but because of the content, the price, and the reliable delivery.

I use “My Yahoo” as my home page, and it hasn’t changed much for a long while until recently.  I attempted to use their beta release and I switched back immediately as it wasn’t ready for prime time.  I’m too busy to devote effort to being a company’s beta tester.  I believe that’s another drawback of Agile: most customers don’t want to be testers.  But here again, the content is way more important to me than the functionality of “My Yahoo.”  

Recently, yahoo revamped their online email application.  Though the features have improved significantly and I’d even say it’s good, I still don’t use web email applications.  I prefer outlook.  I’m not sure why, I just do.

My former company used Concur for employee expense submissions and management. Its first release was a huge improvement over the manual systems of the past.  I used it for a number of years while I was there.  I don’t remember it changing much or at all over those years, but it didn’t need to.  Sure there could have been some UI improvements that would have made it much easier to use, but even if a competitor had a better interface, it would have had to offer something revolutionary to motivate change as the current experience with Concur was invaluable.

I use WordPress for this blog.  I love the application.  They recently released new software, and I promptly upgraded.  I like it, I think it’s better than the older software, but really, I would be satisfied with the old version if they hadn’t offered anything new.  The two most important features for blogging WordPress satisfied long ago.  A blog using WordPress is entirely customizable, and it offers an easy interface for managing content.

What Drives Consumers?

Consumers don’t readily upgrade or replace old products because they have new and improved features.  They replace old products because they have a new need or the upgrade product solves and old problem.  Think about how often you replace physical products in your home because the companies offer products with new and improved features.  How often are you willing to replace your refrigerator, television, stereo, iron, toaster, car, dishwasher, washing machine, stove, and microwave oven, to name a few common home products?  Most of these products are replaced because they are broken, no longer work reliably and/or as well as when they were new.

Summary

The web seemingly changes the economics for companies offering new/improved features because the web has lowered the barriers to delivering changes frequently.  Because the barriers to delivering new/improved features to the consumer are lower, does that mean we should deliver new/improved features more frequently?  What if the software features aren’t the business drivers for your web business?   What’s more valuable to an Amazon.com customer, a new book for sale or a new software feature on the web site?  The danger is that delivering software enhancements frequently gives the illusion that you are delivering more customer value when in reality it’s not the most important need for your customer(s).

The web has changed everything.  The web moved the focus from functionality to data.  Changing software features are not the most important change; it’s changing content that is king on the web.  It’s what I struggle with on my blog and most bloggers struggle with. Do we still need new and improving features on the web? Of course we do, but rate of changing features isn’t nearly as important as it’s argued to be.  Having the right new set of features timely is where the focus needs to be over having any new features quickly. That’s what will win on the web and on the desktop.  Identifying and delivering real customer value is never easy, and it doesn’t get easier with smaller more frequent deliveries.

Agile doesn’t work in the real world is essentially the conclusion of David Starr, an Agile proponent, in his article “Why Agile Doesn’t Work.”  First, he informs us of what is the primary objective of Agile software development methodologies:

Sure, TDD works, so does continuous integration, and a host of other great development practices. The truth is, though, that the real Agile value proposition was never about code. Better software with higher quality and excellent craftsmanship is a great side effect, but Agile is really about changing how products are created and delivered. Agile intends to fundamentally change the model of your relationships with clients and coworkers.

David Starr then proceeds to explain, “Agile supports the idea of frequent delivery of value to customers.”  This is essentially where Agile hits against reality as he further supports in the article.  He identifies two primary reasons why a frequent delivery to customers is problematic in the real world of business. 

1. Once the software exits development, your company’s operations takes too long to move the software to customers.
2. Customers are unable to deploy new versions of software to their users quickly.

His reasons in support of these two points are logical and real.  Read the article; it’s very good.  I make a similar point in my essay “Danger: Agile Practices at Work,” where I also suggest that the goal of frequent delivery of software to most customers is unnecessary.   However, where I disagree is when he suggests that the way to fix this is to have Agile business operations. 

While solving the problem of moving software to the customer faster may help companies that build software, it doesn’t make much sense for most customers to continually have frequent releases of software.  This is true for both companies using software for solutions in their own products, and it’s true for customers that simply use software to run their businesses.

Customer perspective running a business

A successful software product usually solves the primary problems for a majority of customers within the first release or two or maybe three.  For example, let’s examine Microsoft Office suite of applications.   I’ve been using the Microsoft Office suite of applications since the early 90′s when they first became available on Windows.   I basically use the application in the same way as I did with my first purchase around 15 years ago.  I’m not a power user, and I don’t need to be one.  I regularly use 20% or less of the features, and they more than satisfy my need.

Let’s examine another application that I regularly use at home.  I’ve been a Quicken user since the late 80′s when I first purchased it for the Apple Macintosh.   It’s a great application for people who like to keep their checkbooks in balance and manage their stock portfolio.  For the primary consumer purpose of managing financial transactions, Quicken really hasn’t changed much in nearly 20 years nor is change required.

Yes, there have been some improvements along the way that have been valuable in both applications, but most of the time, even with the newest versions, I use the applications as I did when I first purchased them.  For users of these wonderful applications, their objective is to use these applications to solve problems, and if the current version is solving their problems, there really is little reason to purchase the newest version even if it provides more value.  After all, if the current version isn’t solving their problems, then why are they using them?

The only new feature in Microsoft Word that would compel me to purchase the newest version right now is a feature to significantly improve my writing.  For Quicken, there are two features that would compel me to upgrade: one feature is to improve the investment research capabilities.  The other feature is to completely remove the manual entry of downloaded transactions. 

However, even with those features added, price is a factor.  Business and consumers aren’t in business to keep software companies profitable.  They’re not going to purchase every new release even if the value is compelling enough.  Purchasing decisions are always a tradeoff amongst other needs.   Sometimes the reason is only money, other times it’s business priorities, and still other times, the current version simply works good enough for the customer’s needs.

Are the reasons different for software shops?

In my experience never has a naked third party component provided business value to my customers.  If it did, there would be little reason for my development team to exist.   If you’re building a software product, your company has identified an opportunity to add value that either doesn’t exist or value beyond anything out there in the marketplace today.  Third party components add the least value in comparison to the value delivered by your application, so there’s little motivation to upgrade third party components.

When I was writing Windows code for the financial industry throughout the 90′s, very little of our application code was interfacing with the third party components or operating system.   While third party components are important and often necessary, upgrading to the next version of our third party components took time and effort away from delivering additional value to our customers.

It is often rightly a struggle with product marketing to support third party component upgrades in the next release.   Customers don’t purchase applications because they work on the latest version of Oracle or some other third party application that may be part of your product.  Third party components are very often transparent to customers.  They rarely care about what parts are used to assembly your product.    Unless the latest release of the third party component provides some value to the end user of your application, time is better spent investing in customer requirements identified by your product marketing team.

Summary

I suppose David Starr is correct when he says, “Agile is really about changing how products are created and delivered.”  Though, I’m often unsure of what Agile is really about as I often read conflicting opinions from other Agile proponents.   If Agile is “about changing how products are created and delivered,” it’s not a requirement for most customers.

In my product development experience, the next release is mostly about satisfying the following needs:

1. The next release is mostly to expand market share.   It’s about building the new features that will expand the addressable market.  That’s one reason why most users regularly use a small percentage of the delivered functionality in any application.
2. The next release is to support marketing.  It’s about having an answer to your competitor’s features, so when the salesperson pitches the product to a new or old customer, it compares more favorably than your competitor’s product.
3. The next release is to maintain market share.   It’s about giving your customers no reason to evaluate your competitor’s product or even decide to purchase it.
4. The next release is to give your existing customer base a reason to upgrade. It’s about building the features that have additional value for your customers.  However, for the reasons explained above, most of these customers don’t have a need to upgrade as rapidly as is commonly accepted by the Agile community.

David Starr argues that Agile doesn’t work because business operations aren’t Agile, and his remedy is to have Agile business practices that embrace change:  “Embrace continuous integration of the enterprise.”     Customers don’t want continuous change; they want a great product the first time they purchase it, so they can spend more time serving their customers and invest more money on other important needs.  While your product delivery is most important to you and your company, it’s likely not the most important need for your customer.  Agile simply solves the wrong problem.

I’m reminded of an interview I had with a young company for a Software Director role.  They were at point where they were ready to begin rapidly acquiring new customers.  When a new customer signed, there was customization required to have the system operate according to the customer’s demanding requirements.   The turnaround time from signing the new customer and having them productively using the system was approximately four weeks – at least that was the goal. 

The CEO was looking for someone experienced with delivering short release cycles.  His thinking was that a short release cycle had unique characteristics that are not present in long release cycle projects of six to nine month durations.  He identified as one characteristic that there is no room for error in a short release cycle because there is no time to make it up.  With a longer release cycles, of six to nine months, there is slack in the timeline that could accommodate things going wrong without compromising the release date.  Obviously this CEO never worked on long release cycle projects where many are behind schedule before they start.

A four week release cycle seems fit for an Agile approach.  Having short release cycles is one of the Principles behind the Agile Manifesto.  Agile favors an iterative approach to software development, but this release doesn’t require any iterations.  Therefore, it’s not clear to me that Agile is fit for this project since the Agile methodology relies upon the feedback over a series of iterations to arrive at the required solution.  An Agile Methodist might say, “Well, that is easy to solve.  Have a delivery once a week for a total of four iterations.”  Perhaps that would work, but is it enough?

It’s Not The Process

Let’s assume for a moment this problem was one designed specifically for a process methodology like Agile to solve.  Is that all there is to consider?  Assuming a team could deliver successfully to the four week timeline will this be sufficient to address the demands of this growing company?  If the company is signing customers at an accelerating rate, it won’t be long before the backlog of customers grows to unmanageable levels.  The company can hire more people to support the growing demand, but it takes time to hire and train people; therefore, while the team is growing and ramping up to speed, the backlog of work continues to grow.  

Looking to the software development process to solve the delivery requirements of this growing company is a mistaken solution to this company’s aggressive demands, but it’s a solution too often proposed by experienced professionals in the business.  To be Agile in this environment, and all environments that I’ve worked in, requires entirely removing the bottleneck in the software development organization.  The software department needs to find solutions that require less programming, less testing, and less IT personnel – and even none if possible.

Let’s consider some business examples to illustrate the point.  What if the IT department needs to be involved to add a new book, to add a new customer, or to change an item price on Amazon.com?  Would the software development process be fast enough for Amazon.com to remain competitive?  Or let’s take a web hosting company, what if the IT department needed to get involved for each new customer that signs up? Would the software development process be sufficient for any web hosting company to remain competitive?  What if every blogger using Worpress needed to go to WordPress.com for custom site designs?  Would the software development process be sufficient for WordPress to remain competitive?   Could WordPress ever adequately meet its customer’s myriad and demanding needs if the skin was not customizable by the end user?   Would WordPress have as large a market share as it has now if the software was not end user customizable?

To be more agile, the IT department needs to rethink its role in the organization.  The role isn’t to be the high priests of some complicated methodology or technology, but their role is to identify the technologies and methodologies that permit the company to compete in the marketplace more efficiently and strategically while reducing costs and time to market.  It’s not to apply the hammer to every problem, but to create the tools, technologies and/or processes that are appropriate for each and every unique business requirement.

Where To Focus

While emphasis on software development methodology will never go away and shouldn’t, the software process methodology is not where the IT department can significantly reduce costs and time to market, and increase agility.  Every study has confirmed that effort and software size are directionally correlative: the more code that is written, the more effort that is required to complete the delivery.  Assuming we have a problem where the best solution requires one million lines of code to complete and all the requirements are needed to be delivered day one to have a marketable product, what option will reduce costs and time to market, and increase agility the most?  Will it be choosing Agile over Waterfall or will it be finding other ways to solve this problem?  For all but the most naïve, finding another way to solve this problem will yield the largest returns.

All the significant gains in software development productivity have come from improvements in technology.  Let’s examine some of them. 

1. The introduction of high level languages significantly improved programmer’s productivity where one high level language statement translates into many lines of assembly language.  Assuming that programmer productivity is 10 lines of code per day, writing 10 lines of high level language code will yield more functionality than 10 lines of assembly language code, thus the productivity of functionality delivered has increased.  In addition, the improved readability of high level languages improved the ability to understand the implementation details faster.  A quicker learning curve makes the software easier to maintain.
2. Object oriented languages significantly improved productivity over procedural languages such as C or Pascal.  With an object-oriented language you can reuse code easily.  Templates and inheritance provide for reuse in significantly improved ways.  Solutions are better abstracted: they more closely model the problem domain.
3. Standard libraries, design patterns, and standard protocols significantly improved productivity for a number of reasons.  One, there is no need to reinvent the wheel reducing the effort in design and implementation.  Two, the solutions have been tested and confirmed to be appropriate by other experiences.  Three, there is a rich supply of experienced professionals with knowledge of the technologies to draw upon.
4. Software development tools, such as defect tracking software, integrated development environments, software debuggers, configuration management software, compiler technology, and even syntax aware editors to name a few, have all improved the software developer’s and the team’s ability to build software faster and more reliably.

How much longer do you think a release cycle would be if we didn’t have these technological advances?   What if we were still coding our software in assembly language?  What if we were still managing our defects in loose leaf binders as I did early in my career?  What if we didn’t have the advantages of feature rich libraries and foundation classes?  If we used Agile to solve our problems using the old technologies, would we see as much improvement as we have with these technology advances?

Being Agile

Like the productivity gains provided to software development teams by the advances in software development tools, we can improve our productivity with similar advances to our business problems.  We need to understand the essence of the customer need and relinquish control to the customer, customer teams, and/or business partners.  Once we understand the customer’s changing needs, we can architect the solution in a way that supports making those changes easier, faster, and with higher quality.   This may involve one or more of the following approaches

• Provide a messaging protocol in a standard format such as XML.
• Provide an API to the functionality that a customer may want to reuse in other ways or to customize the application such as done with WordPress.
• Provide an editing interface to the application for the features that are likely to require changes by the customer.  For example, if the system is rules driven, provide the user with the ability to edit rules for themselves.  Make the rules data driven, don’t hard code the rules in whatever chosen language the system is implemented.
• Provide tools that change the behavior of the system.  Some of these tools can be pushed out to the end users, customer teams, and even business partners.  In other cases, the tools would generate the code that needs to be built by the development team.
• Provide a rich architecture that more closely mirrors the problem domain, is easily extensible, and is orthogonal.

These are only a few of the many possibilities that are available to us.  In many cases, the possibilities are unique to the domain.  Some of these options need to be attenuated by the life expectancy of the product as providing these capabilities require an investment, and the return may not support the investment with a product having a short life expectancy.

Delivering architecture and tools with these capabilities requires a large investment in up front analysis and design.  Developing these solutions often requires a long development lead time before the first build is available for test – it’s typically longer than a few months.  What you give up on the front end, though, you gain in leaps and bounds in the back end.  After all, if we are to believe the tired maxim that 80% of the effort is spent in maintenance, doesn’t it make sense to design solutions that will optimize the maintenance phase?

Conclusion

The Agile methodologies are not suited for delivering solutions of this nature.  An emergent, iterative approach does not have much to offer as a solution when the requirements for solving the problem require a large up-front investment in analysis and design.  The Agile methodologies are designed to solve today’s problems instead of anticipating tomorrow’s problems and instead of developing architectures and tools that anticipate them.  Agile approaches favor delivering something quickly at the expense of a well thought out architecture and at the expense of anticipating future requirements. 

While Agile with a Big “A” is a development process, being agile has little to do with the development process.  Improving a company’s ability to be agile, to deliver quicker, and to deliver higher quality requires the IT department to take an even broader perspective.  They need to look at the end-to-end business processes, develop tools and processes that leverage the corporate resources to their fullest in the interest of delivering the greatest value to the customer and to the shareholders.  Optimizing the software development methodology to achieve agile goals is too narrow a perspective to solve this problem.  Of course, we should improve the Software Development Processes, not so much to be agile, but to be more efficient, to be repeatable, to improve quality, and to make better decisions.  Agile as a solution for being agile is a misnomer.

One of the difficulties I have with the Agile software methodologies and its proponents is that they go to far.  They are often susceptible to hyperbole while hawking their methodology like used car salesmen.   Their embracing of the practice of refactoring is one example of this.

Of course, we should refactor, but refactoring isn’t a design methodology,  it isn’t a way to be agile, and it doesn’t help deliver software faster.  Refactoring as advocated by the Agile processes is essentially a design methodology.  A team following Agile arrives at an optimal design and implementation via a series of iterations and refactoring efforts.  Refactoring as a design methodology is incompatible with the Agile goals of delivering faster and accommodating change anytime in the project life-cycle.

Refactoring is necessary, the Agile proponents say, because it is impossible to know enough about the system up front to arrive at a good enough up front design.  In fact, Agile is in part a reaction to the Big Up Front Design approach embraced by Waterfall adherents.  The Agile camp has essentially given up on the goal of arriving at an optimal up front design and instead relies on refactoring at every iteration until an optimal design is achieved.

Let’s think about this approach for a moment.  There is a cost — maybe even significant cost — associated with refactoring.   It squanders time away from the schedule that can better be spent developing new features. 

While refactoring is change, it isn’t the agile change that was intended: it’s change that involves redoing work that essentially delivers no new features to the customer.   This rework is encouraged to be repeated multiple times throughout the product life-cycle. 

It costs time and money to constantly rework code while stealing time and resources from delivering additional value to the customer.  Further, with each refactoring effort the code base is destabilized adding new defects and temporarily lowering quality of the software that was already tested and delivered to customers.

One would believe that an approach with the goal of being agile would attempt to minimize the amount of refactoring, but they don’t: they encourage it.  It is a mistaken belief of the Agile proponents that it is impossible to arrive at an elegant, orthogonal, extensible design up front.  My experience is that it is possible to arrive at an optimal design up front, and when you do, the benefits to the product are immense.  Benefits include accelerated time to market, high levels of quality, and shorter learning curves for new employees and lower costs over the product lifetime.

Orthogonality is a characteristic of a good design and even a good process.  The assembly line process had its birth in the auto industry, yet it has been applied successfully to a myriad of industries, demonstrating its orthogonality.  When a design or process is orthogonal it isn’t confined to a small domain of applicability.  If you’ve ever worked as a cook in McDonalds, you would have witnessed assembly line practices successfully at work.   The Waterfall software life-cycle is another example of the assembly line process successfully at work.

The Agile practices, on the other hand, are not likely to be adopted by many other industries.   Arriving at an optimal implementation over many iterations and rework is too costly in time and resources to be adopted successfully by other industries.   Imagine building a house, or any structure, with an Agile approach.  The time and resources and consequently costs would be too great with building a part of the structure, tearing it down, and reassembling it differently to arrive at the desired architecture when it could have been discovered up front with more investment in requirements and pencil and paper design.

The Agile practices give the illusion of faster, more, and cheaper, but it’s only that: an illusion.  It achieves this legerdemain by delivering more releases of shorter duration with less functionality.   In a time when society is looking for easy answers to difficult problems while embracing the appearance of success, the Agile practices are the right practices for our troubled times.

When I was a high school student I was an avid competitor in the sport of wrestling.  It’s an extremely demanding and punishing sport: requiring extreme stamina, strength, skill, agility, and mental toughness.  You have to have a strong mind to compete successfully in wrestling.  When you’re in the 3rd period of a match, you’re exhausted, and your opponent continues aggressively to push the action, only the tough-minded continue to fight and pull out a win.

Preparation for competition is key and demanding.  Practices are lengthy, exhausting, painful, and mentally draining.  It is common for a wrestler to lose anywhere from 3 to 10 pounds of body weight in a single practice.  Preparation requires discipline to keep with a training and weight loss program.  The diet can drain an athlete mentally having to refrain from the daily temptations of satisfying treats.

At times even the best competitors hate the sport, yet they push themselves to endure the challenge for they know to be the best requires intense preparation.  There are no shortcuts to success, not even natural talent.  Elite wrestlers would never allow their coach to run anything less than a demanding practice.  They seek out coaches who will push them to their limits.

The software community can learn much about dedication and commitment to excellence from wrestlers.  A wrestler would never tolerate shortchanging the “Big Upfront” work required for success as we do in the practice of software development — no matter how painful and difficult the upfront work.   Instead of arguing for dedication to the “Big Upfront” work, proponents of Agile practices argue against it.

Just as in wrestling, it’s been my experience that a commitment to the “Big Upfront” practices makes for highly successful software projects.  “Big Upfront” has become an anathema in the software community dedicated to mediocrity, chaos, and low quality.  Because it is difficult, it is often neglected. 

We often forget that we are paid to do a job, and it requires us to do what’s required, not just what we like to do.   Having some talent and in demand skills does not give us any special privileges, though people often behave as it does.  Eventually, the issues making the “Big Upfront” work challenging come to haunt you in the project when skipped.  There is no escaping it; it’s only postponed.

Like the wrestler preparing for competition, we need to embrace the challenges in the “Big Upfront” work required to deliver successfully on a project even though it may be painful.   When a wrestler fails in his quest, he’ll often rightfully blame a lack of sufficient preparation, but when a software team fails, they often fail to see the inadequate preparation as a factor.

A commitment to excellence requires no less than a complete commitment to doing what must be done in spite of all the pain and obstacles.  If achieving excellence were easy, it wouldn’t be so uncommon.  The irony is that embracing the “Big Upfront” work actually makes the project easier – not harder.  Elite athletes know and accept this truth.  It’s about time the software community does too.  Only when we do, can we begin to advance the practices of managing software development.

The Agile software development practices are in their infancy stage as evidenced by the number of variants that are being promoted in popular print and usage today:  Scrum and XP being two popular variants.  Clearly the Agile practices are still evolving, and I believe what will eventually emerge are a set of Agile practices that look awfully familiar to traditional approaches.  The question is, is once the Agile community arrives there, will the conflict cease?

My expectation is that it will not.  As the Agile approach converges with traditional approaches, the Agile zealots will declare that they’ve been abandoned by name the popular evil – corporate types maybe.  The other more important reason that the debate will fester is software projects will continue to fail with both approaches.  This fact will only continue to fuel the Agile zealots’ fire that they’ve been abandoned, and if teams would only adhere more rigorously to the pure Agile tenets, projects would not fail.

Reading Wars 

Much of the friction between the Agile camp and the Traditional camp is reminiscent to the Reading Wars that are being waged across America and internationally.  The traditional approach to teaching children to read advocates a phonics approach.  After all, the written alphabet was designed to encode sound.  It only stands to reason that to read requires a person to learn the encoding rules and to decode the written word.  That’s how I learned to read.  It certainly works.

The modern advocacy to teaching children to read is with a process called Whole Language (don’t the issues sound a lot like Agile?).   The theory is that reading should evolve from print like speaking evolves from hearing.  No one instructs a child directly with the rules of the spoken word.  Children learn to speak through trial and error while immersed in spoken language.  Likewise, immerse children in the printed language, and children will learn to read almost intuitively.  It sounds nice on paper.

One of these two approaches to teaching children to read is the best approach.  Where the inferior approach has got it all wrong is that failure for children to learn to read well has nothing to do with the approach that was replaced.  The child’s God given ability is a more important factor in success or failure to read well, and so we argue and debate about the reading process that had nothing to do with failure.  A good process ends up being replaced with an inferior process contributing to more reading failure, but the people cornering the debate are unwilling to explore the possibility that they’ve introduced something that doesn’t work.

Similarities 

In many ways the issues in the Reading Wars parallel the Software Process Wars:

• Children have learned to read with both approaches, and software projects have been delivered with Agile and Traditional approaches. This is why the debate never ends.

• Children who are the most successful readers with the Whole Language approach are the smartest children. Agile proponents require the team to be highly capable for success. Having highly capable people on a team is great and important, but it’s impossible to build an organization with only extremely talented people.

• Children fail to read well with both reading approaches. Software projects fail with both Agile and Traditional approaches. Sometimes failure has nothing to do with the process, and sometimes it has everything to do with the process.

• The Whole Language approach is built upon a flawed premise: decoding skills do not need to be taught directly; they evolve through immersion in the printed word. The Agile approach is based on a false premise: waterfall phases can be ignored and don’t apply.  A flawed theory is normally built upon an incorrect premise.

• Whole Language success requires higher skilled teaching professionals.  Agile success requires higher skilled engineers and management.  It’s impossible to build an entire organization with only top talent, and it is not even advisable.  A mixture of ability makes for a succesful team; though, having no standard for minimum ability is bad.

Conclusion

Fundamentally, the reason the Reading Wars and Software Process Wars will continue unabated is there will always be failure with either approach.  Where the promoters of the inferior software paradigm have it all wrong is that in many cases failure has nothing to do with the process, it has everything to do with the quality of the people leading and working the project and organization.   People issues are difficult problems to fix and to talk about, so people debate what’s easier to talk about and change.  In fact, many people recommend new processes as a solution to fix people problems when that only seems to exacerbate problems as I can’t recall a case where a change in process made things better when the wrong person was responsible for the job.

It is my plans to discuss the people and management issues that impede project and team success.  I don’t profess to have all the answers, but I plan to present what’s worked well for me and what I’ve witnessed work well by the people I’ve reported to.   Some of the topics I plan to touch upon are organization structures, rewards, making change happen, and performance feedback to name a few.

Have you ever wondered why software process has yet to flourish in the software industry?  Why, after many decades of industry growth, there is no consensus on a process methodology or even best practices?  Why specious Agile approaches have captured significant mindshare in the software community?  Why when you mention the words software process, software professionals cringe?

In other industries process innovation has been the contributing factor in the explosive growth.  Where would the industrial revolution be without the innovation of assembly line processes?  Where would the auto industry be?  A predictable, repeatable process has been the cornerstone of innovation and quality for many successful industries.  Did you ever wonder why a hamburger and french-fries at McDonalds taste the same no matter where in the world you purchase it?  It’s all about process.

It’s an enigma to me and others who have had great success with software process that we continue to debate the need for formal process in the software industry.  Yet the software industry continues unabated with a record of late deliveries, budget overruns, missed expectations, and low quality.  In this article, I’d like to explore the factors that contribute to software process adoption failure.

Undisciplined

People are undisciplined by nature.  It requires effort to be disciplined, and you usually need to be disciplined for things you need to do but find hard to do.  For example, how many of us can stick with a regular exercise or diet program?  Both activities are great for our health. It’s difficult to argue against the need for discipline to be successful in those healthful endeavors, yet most of us find it difficult to stay with an exercise or diet routine for long.  Similarly, software process is difficult to stick with, though it benefits our projects.  Without discipline, software process is sure to decay and ultimately to fail.  Reward, hire, and promote people who demonstrate discipline.  The most successful people demonstrate the highest degree of discipline.

Bureaucracy

Many Software Processes are overly bureaucratic. I’ve noticed a strange phenomenon when people are given the assignment to develop a process area: they behave as if they are writing a Ph.D. thesis.  They create an elaborate and complicated solution when a simpler but appropriate solution would suffice.  My former supervisor would often quote, “Process should be like a woman’s skirt: short enough to be interesting, long enough to cover the subject.”  Process should be lean and mean.  Bureaucracy is an unwelcome friend of process.  This is a behavior that one needs to be continually vigilant in preventing to develop process that succeeds.

Ineffective Process

Some processes, maybe even many, aren’t good.  Just because you have invested a great deal of time developing and deploying your process, it does not mean it is effective or good – even if you were successful in achieving an audited process level.  If you are not getting the benefits from your process that you expected, think about reassessing whether you’ve developed and deployed an effective process. Most importantly, fix it.

Management Priority

Management needs to show by example that they believe process is important. Employees take serious what management demonstrates to be important.  As identified earlier, process requires discipline from the team.  If management isn’t making sure that the process is being followed, the discipline will wane.  If you are a manager and you don’t believe in process, don’t even start.   You’re only wasting the organization’s money and time.

Management Commitment

It takes commitment to be successful with process.  Gaining acceptance is often challenging because people find it difficult to change their practices – even if the change promises to improve their working conditions.  Don’t expect a good process to be introduced in less than 18 months:  normally teams are juggling their current project commitments with activities to deploy a new process.  A comfortable measured pace is what’s required.  After the project is rolled out and fully deployed, it takes continued commitment to sustain the process.   This is very much like starting an exercise program to lose weight.  It takes time before you start seeing results, and once you reach your goal, it takes continued effort to maintain the weight loss.  I would guess that about 50% of organizations that failed, never reached their process goal, and the other half quit during maintenance.

Misused

Often time software teams find themselves committing to extremely aggressive deliveries, and they misuse process as a way of slowing the business groups down.  When teams do this, you’ll hear comments like the process says we don’t deliver estimates until the requirements have been signed off or similar types of arguments that delay the development team from making a commitment.  This is the surest way to undermine the business group’s support for process in your organization.  When the business group doesn’t get the cooperation and support that it deserves, they will eventually withdraw their support for your process.  Not only will this behavior result in the abandonment of the process, but it will impede the software group’s ability to manage the work load more effectively, as a good process is the only way to achieve that.

Overemphasize Process Levels

When organizations embark on process initiatives, management will often set a deployment goal.  For teams using CMMI as the model, the goal will be to reach level 2 by a given date.  This is the wrong emphasis as reaching a process level signifies nothing.  Like any certification, reaching a process level only signifies that you’ve met the requirements of the discipline.  It has loose correlation to process effectiveness. The reason to deploy process is to improve your organization.  Have measures for demonstrating improvement, and let those measures be the primary goal. 

I don’t intend to suggest there is no value in achieving process level maturity.  It’s the emphasis that concerns me.  It’s easy for teams to game the system and reach a maturity level when they really aren’t following much of the process that they present to the audit committee.  If the emphasis is on tangible results, there is less reason to game the system.

Dedicated Process Groups

It’s natural for organizations to embark on a process initiative by creating a process group staffed with full time process engineers.  While the effort required to build and roll out process in an organization is a great deal of work, too often this is the least effective approach to rolling out software processes for many reasons.  There are two that make it a particularly incorrect approach.  One is that often dedicated process teams are staffed with people who have never been successful in managing the delivery of a product themselves.  On paper delivering software looks like a simple endeavor, but it’s impossible to get a feel for the nuances of leading and managing a team from a book.  You need to experience it.  Weak, inexperienced engineers develop unproductive processes. The second drawback of a dedicated process group is that the process group’s self-esteem is tied to rolling out process, so they tend to document and develop many procedures to be followed.  Before you know it, you have layers upon layers of bureaucracy for delivering a single line of code.  It’s better, yet more challenging, to have process teams staffed with employees who have a part time responsibility for creating and deploying the process.

Poor Process Audits

As identified earlier, good process requires discipline.  Audits are necessary to enforce discipline. However, there are some pitfalls.  Auditors may work with the people they audit everyday, and in this case they may find it uncomfortable writing up a negative audit report.  One way to combat that is to have formal audit objectives with specific pass or fail criteria.  This removes the auditor from making a subjective call on the audit.  The only question the auditor should set out to answer is, is the team following the documented process?

Doesn’t Promote Change

The only ideal process is a process that changes.  Like a software release, the first version of your process will be buggy.  You’ll need to tweak it, and maybe even overhaul parts of it.  As the team becomes more experienced with process, they will find the practices that they originally thought would be effective can be improved. The process needs to encourage change, and it needs to be easy to change.  People will do what works, so it’s important to make it easy to change; otherwise, the documented process will not reflect what people actually do.

Fails to Demonstrate Benefits

Teams embarking on process often overlook ensuring that the process demonstrates benefits to the organization.  The business benefits must be one or more of the following: improved time to market, increased productivity, reduced costs, improved quality, reliable and predictable commitments, or improved customer satisfaction.  If there is no demonstrable benefit, why continue to do it, and why should management continue to sponsor it?

Been There Before

For the seasoned employee, chances are they’ve been through this exercise one or more times in their career, and usually the experience isn’t positive: management removes its sponsorship as things were actually starting to improve. Consequently, people are reluctant to invest in something that management won’t sustain. People become cynical about these initiatives, and they come to know if they hold management off long enough, they will abandon the initiative.

Companies Are Too Dynamic

The organization structure in companies today is too dynamic.  Way before a process has a chance to be institutionalized, the key sponsors are off to new assignments, or the teams are broken up and reassembled offshore.   It’s important to have organizational stability to nurture culture change in an organization.  This one is a tough one to solve.

Business Groups Are Uncomfortable

It’s difficult to get the business groups on board with process.  They are often satisfied  to manage release requirements informally.  They often fail to appreciate the need to prioritize requirements and create product roadmaps.  They struggle to identify release commitments with anything less than everything.  When you attempt to socialize the product group into the new software process, they feel constrained, and when they think the process is the reason they aren’t getting what they feel they need, they kill the process initiative.

Summary

There are many challenges to overcome when rolling out software process in a company.  Adopting and rolling out process shares many characteristics with keeping to a diet program.  They’re both good for you, they require discipline to be successful, there are many forces working to derail you, and there are many fad programs.  The current process fad is the Agile methodologies.  Like the fad diet program that promises fast weight loss with no discipline, Agile practices promise faster releases, higher quality, and more satisfied customers with less discipline.   And like the dieter eager for a quick, painless weight loss program, the software community is eager for a silver bullet.

A former boss of mine had a sign hanging on his wall that read “good software takes time.”   Likewise, good process takes time: time to develop, time to rollout, time to perfect and time to institutionalize.  It requires discipline, commitment, experience, and savvy to develop and maintain.  If you are embarking on a software process initiative or you are dissatisfied with the progress of your current initiative, it would be worthwhile to review the challenges identified in this article and develop a mitigation strategy for the items you believe may impact your program.  For those who are prepared and eager for the challenge, the rewards of successful process implementation are great.

Further Reading

• CMMI(R): Guidelines for Process Integration and Product Improvement (2nd Edition) (The SEI Series in Software Engineering)
• Real Process Improvement Using the CMMI
• Agile Project Management with Scrum (Microsoft Professional)
• Agile Software Development, Principles, Patterns, and Practices
• Extreme Programming Explained: Embrace Change (2nd Edition) (The XP Series)
• Extreme Programming Installed (The XP Series)
• Systematic Process Improvement Using ISO 9001:2000 and CMMI(sm)
• ISO 9001, The Standard Interpretation
• ISO 9001 Survival Guide, Third Edition