Have you ever wondered why software process has yet to flourish in the software industry?  Why, after many decades of industry growth, there is no consensus on a process methodology or even best practices?  Why specious Agile approaches have captured significant mindshare in the software community?  Why when you mention the words software process, software professionals cringe?

In other industries process innovation has been the contributing factor in the explosive growth.  Where would the industrial revolution be without the innovation of assembly line processes?  Where would the auto industry be?  A predictable, repeatable process has been the cornerstone of innovation and quality for many successful industries.  Did you ever wonder why a hamburger and french-fries at McDonalds taste the same no matter where in the world you purchase it?  It’s all about process.

It’s an enigma to me and others who have had great success with software process that we continue to debate the need for formal process in the software industry.  Yet the software industry continues unabated with a record of late deliveries, budget overruns, missed expectations, and low quality.  In this article, I’d like to explore the factors that contribute to software process adoption failure.

Undisciplined

People are undisciplined by nature.  It requires effort to be disciplined, and you usually need to be disciplined for things you need to do but find hard to do.  For example, how many of us can stick with a regular exercise or diet program?  Both activities are great for our health. It’s difficult to argue against the need for discipline to be successful in those healthful endeavors, yet most of us find it difficult to stay with an exercise or diet routine for long.  Similarly, software process is difficult to stick with, though it benefits our projects.  Without discipline, software process is sure to decay and ultimately to fail.  Reward, hire, and promote people who demonstrate discipline.  The most successful people demonstrate the highest degree of discipline.

Bureaucracy

Many Software Processes are overly bureaucratic. I’ve noticed a strange phenomenon when people are given the assignment to develop a process area: they behave as if they are writing a Ph.D. thesis.  They create an elaborate and complicated solution when a simpler but appropriate solution would suffice.  My former supervisor would often quote, “Process should be like a woman’s skirt: short enough to be interesting, long enough to cover the subject.”  Process should be lean and mean.  Bureaucracy is an unwelcome friend of process.  This is a behavior that one needs to be continually vigilant in preventing to develop process that succeeds.

Ineffective Process

Some processes, maybe even many, aren’t good.  Just because you have invested a great deal of time developing and deploying your process, it does not mean it is effective or good - even if you were successful in achieving an audited process level.  If you are not getting the benefits from your process that you expected, think about reassessing whether you’ve developed and deployed an effective process. Most importantly, fix it.

Management Priority

Management needs to show by example that they believe process is important. Employees take serious what management demonstrates to be important.  As identified earlier, process requires discipline from the team.  If management isn’t making sure that the process is being followed, the discipline will wane.  If you are a manager and you don’t believe in process, don’t even start.   You’re only wasting the organization’s money and time.

Management Commitment

It takes commitment to be successful with process.  Gaining acceptance is often challenging because people find it difficult to change their practices - even if the change promises to improve their working conditions.  Don’t expect a good process to be introduced in less than 18 months:  normally teams are juggling their current project commitments with activities to deploy a new process.  A comfortable measured pace is what’s required.  After the project is rolled out and fully deployed, it takes continued commitment to sustain the process.   This is very much like starting an exercise program to lose weight.  It takes time before you start seeing results, and once you reach your goal, it takes continued effort to maintain the weight loss.  I would guess that about 50% of organizations that failed, never reached their process goal, and the other half quit during maintenance.

Misused

Often time software teams find themselves committing to extremely aggressive deliveries, and they misuse process as a way of slowing the business groups down.  When teams do this, you’ll hear comments like the process says we don’t deliver estimates until the requirements have been signed off or similar types of arguments that delay the development team from making a commitment.  This is the surest way to undermine the business group’s support for process in your organization.  When the business group doesn’t get the cooperation and support that it deserves, they will eventually withdraw their support for your process.  Not only will this behavior result in the abandonment of the process, but it will impede the software group’s ability to manage the work load more effectively, as a good process is the only way to achieve that.

Overemphasize Process Levels

When organizations embark on process initiatives, management will often set a deployment goal.  For teams using CMMI as the model, the goal will be to reach level 2 by a given date.  This is the wrong emphasis as reaching a process level signifies nothing.  Like any certification, reaching a process level only signifies that you’ve met the requirements of the discipline.  It has loose correlation to process effectiveness. The reason to deploy process is to improve your organization.  Have measures for demonstrating improvement, and let those measures be the primary goal. 

I don’t intend to suggest there is no value in achieving process level maturity.  It’s the emphasis that concerns me.  It’s easy for teams to game the system and reach a maturity level when they really aren’t following much of the process that they present to the audit committee.  If the emphasis is on tangible results, there is less reason to game the system.

Dedicated Process Groups

It’s natural for organizations to embark on a process initiative by creating a process group staffed with full time process engineers.  While the effort required to build and roll out process in an organization is a great deal of work, too often this is the least effective approach to rolling out software processes for many reasons.  There are two that make it a particularly incorrect approach.  One is that often dedicated process teams are staffed with people who have never been successful in managing the delivery of a product themselves.  On paper delivering software looks like a simple endeavor, but it’s impossible to get a feel for the nuances of leading and managing a team from a book.  You need to experience it.  Weak, inexperienced engineers develop unproductive processes. The second drawback of a dedicated process group is that the process group’s self-esteem is tied to rolling out process, so they tend to document and develop many procedures to be followed.  Before you know it, you have layers upon layers of bureaucracy for delivering a single line of code.  It’s better, yet more challenging, to have process teams staffed with employees who have a part time responsibility for creating and deploying the process.

Poor Process Audits

As identified earlier, good process requires discipline.  Audits are necessary to enforce discipline. However, there are some pitfalls.  Auditors may work with the people they audit everyday, and in this case they may find it uncomfortable writing up a negative audit report.  One way to combat that is to have formal audit objectives with specific pass or fail criteria.  This removes the auditor from making a subjective call on the audit.  The only question the auditor should set out to answer is, is the team following the documented process?

Doesn’t Promote Change

The only ideal process is a process that changes.  Like a software release, the first version of your process will be buggy.  You’ll need to tweak it, and maybe even overhaul parts of it.  As the team becomes more experienced with process, they will find the practices that they originally thought would be effective can be improved. The process needs to encourage change, and it needs to be easy to change.  People will do what works, so it’s important to make it easy to change; otherwise, the documented process will not reflect what people actually do.

Fails to Demonstrate Benefits

Teams embarking on process often overlook ensuring that the process demonstrates benefits to the organization.  The business benefits must be one or more of the following: improved time to market, increased productivity, reduced costs, improved quality, reliable and predictable commitments, or improved customer satisfaction.  If there is no demonstrable benefit, why continue to do it, and why should management continue to sponsor it?

Been There Before

For the seasoned employee, chances are they’ve been through this exercise one or more times in their career, and usually the experience isn’t positive: management removes its sponsorship as things were actually starting to improve. Consequently, people are reluctant to invest in something that management won’t sustain. People become cynical about these initiatives, and they come to know if they hold management off long enough, they will abandon the initiative.

Companies Are Too Dynamic

The organization structure in companies today is too dynamic.  Way before a process has a chance to be institutionalized, the key sponsors are off to new assignments, or the teams are broken up and reassembled offshore.   It’s important to have organizational stability to nurture culture change in an organization.  This one is a tough one to solve.

Business Groups Are Uncomfortable

It’s difficult to get the business groups on board with process.  They are often satisfied  to manage release requirements informally.  They often fail to appreciate the need to prioritize requirements and create product roadmaps.  They struggle to identify release commitments with anything less than everything.  When you attempt to socialize the product group into the new software process, they feel constrained, and when they think the process is the reason they aren’t getting what they feel they need, they kill the process initiative.

Summary

There are many challenges to overcome when rolling out software process in a company.  Adopting and rolling out process shares many characteristics with keeping to a diet program.  They’re both good for you, they require discipline to be successful, there are many forces working to derail you, and there are many fad programs.  The current process fad is the Agile methodologies.  Like the fad diet program that promises fast weight loss with no discipline, Agile practices promise faster releases, higher quality, and more satisfied customers with less discipline.   And like the dieter eager for a quick, painless weight loss program, the software community is eager for a silver bullet.

A former boss of mine had a sign hanging on his wall that read “good software takes time.”   Likewise, good process takes time: time to develop, time to rollout, time to perfect and time to institutionalize.  It requires discipline, commitment, experience, and savvy to develop and maintain.  If you are embarking on a software process initiative or you are dissatisfied with the progress of your current initiative, it would be worthwhile to review the challenges identified in this article and develop a mitigation strategy for the items you believe may impact your program.  For those who are prepared and eager for the challenge, the rewards of successful process implementation are great.

Further Reading

• CMMI(R): Guidelines for Process Integration and Product Improvement (2nd Edition) (The SEI Series in Software Engineering)
• Real Process Improvement Using the CMMI
• Agile Project Management with Scrum (Microsoft Professional)
• Agile Software Development, Principles, Patterns, and Practices
• Extreme Programming Explained: Embrace Change (2nd Edition) (The XP Series)
• Extreme Programming Installed (The XP Series)
• Systematic Process Improvement Using ISO 9001:2000 and CMMI(sm)
• ISO 9001, The Standard Interpretation
• ISO 9001 Survival Guide, Third Edition