There’s a lot of activity and promotion of the Agile software methodologies that can be found on both the Internet and in books today.  Its adherents are zealous in their commitment to the Agile principles codified in the principles behind the Agile Manifesto.    I first became interested in the Agile development practices when the Extreme Programming practices began generating a lot of interest in the software community.  My first reaction to this was to be turned off.  The name conjured up images of chaos, cowboy programmers, and death march development projects — practices I am all too familiar with.  I found the advocacy of pair programming to be unnatural and impractical.

After reading many articles published by Agile proponents, I remain steadfast in my beliefs that something is wrong, and rather than advancing the state of the art practices in Software Development, Agile proponents are setting us back, and in this world where jobs easily cross international boarders, especially easy in software, we need practices that demonstrate the value proposition for keeping teams here in the US: practices that deliver reliably and predictably on commitments and practices that can demonstrate improvements in productivity and quality.

Wikipedia explains, the Agile practices developed out of a reaction against “heavyweight” methods.  The Agile community has some valid gripes with traditional practices.  Traditional practices have had their share of problems, but are the solutions advocated by the Agile community a step in the right direction?  I believe not, and I’d like to compare and contrast some of the problem practices of traditional methodologies against the practices advocated by Agile methodologies.

Overboard With Documentation

Traditional practices often go overboard in requiring documentation, often producing many pages no one ever reads or uses.   That needs to be corrected.  The Agile proponents promote oral communication over documentation.  Oral communication is vital.  There is nothing like the fidelity that can be had in a good conversation and it saves time.  However, there is tremendous value in formal communication - especially having written requirements.  Documentation frees people’s minds for new information.   Memories often fail and people never remember every detail quite like it was intended.  Put two or more people in a room to agree on requirements, and each will remember some of the details of the requirements differently.  Further, formal requirements allow other people to review the details and provide further input.  Reviews are a best practice when done intelligently.  There is no opportunity for review without a written record.

Resist Change

Traditional practices often resist change — especially major requirement changes after the requirements have been signed off.  The practice sometimes leaves the business group feeling that the software team doesn’t share the same goals as the business.  The software team gets a reputation for delivering less than what was expected.  That’s never positive for the software team.  The Agile proponents promote and value continuous change, anytime in the life cycle.  But too much change is inefficient, expensive, and counter productive to progress and team morale.   I’d love to know what applications Agile practitioners are building because in my 20+ year’s experience, requirements rarely changed significantly on my projects. Instead, they grew.  When requirements are changing too much and too broadly, it’s a signature of a project that is ill conceived and lacks good leadership.  This is nothing new, and it has been the cause of more project failures since the beginning of software development.

Rigid Project Life-cycle

Traditional practices have promoted a sequential phase strategy often requiring one phase to complete in its entirety before beginning the next phase.   This is a very rigid, inefficient, and tedious approach to software development.  The Agile proponents have promoted more of a parallel delivery strategy.  Some proponents of Agile methodologies seem to deny the existence of dependencies and the existence of a natural waterfall approach to software development.  The reality is that there is a natural waterfall behavior to the software development life-cycle: there is no avoiding the natural progression of requirements, design, code, and test phases of software development.  You can’t test what hasn’t been built, and you can’t code what hasn’t been defined.   While some traditional practitioners have followed a rigid waterfall approach, most traditional practitioners maximize parallelism.  It is difficult not to when the time pressures are omnipresent.

Big Bang Approach

Traditional methodologies have promoted a big bang integration and test strategy.  While it is commonly accepted that big bang integration approaches have major drawbacks, it is likely that teams will continue to engage in this approach, and in some very big projects it may be unavoidable.  The Agile proponents have promoted continuous test and short incremental deliveries.   Incremental deliveries are a best practice that traditionalists embrace.  Where the Agile community has it wrong is that the increments are normally time-boxed. Many deliveries require longer durations than the short time intervals advocated by Agile practitioners.    Traditional practitioners let the delivery dictate the interval as it can only be accomplished.

Upfront Definition

Traditional methodologies have relied upon considerable up front definition and design.  When done well, the applications are built upon solid architectures that accelerate future deliveries. Agile methodologies promote an evolutionary approach to definition and architecture believing it’s not possible to identify accurately up front enough of the requirements and architecture for constantly changing customer requirements.  The mantra in the Agile community is constantly refactor.  This is a highly expensive approach, and in this age of globalization where costs drive decisions, it should be looked at with trepidation.  This isn’t a wise approach, as market pressures to deliver quickly always constrain your latitude for refactoring.  If it’s a choice between delivering something two months late to improve architecture and delivering early with the old architecture, most times delivering early wins at the expense of further architecture decay.   Constant refactoring is expensive.  You’ll often hear teams in the trenches lament about how it’s too expensive to do it right the first time but it’s never too expensive to do it over, and in the case of Agile, over, and over, and over, and over…

Infrequent Large Deliveries

Traditional methodologies tend to deliver new functionality to customers less frequently, and they consequently have the reputation of being unresponsive to customers.  Agile proponents promote frequent releases from “a couple of weeks to a couple of months.”  Most customers will not adopt such rapid changes.  It costs time and money to deploy and train people on any new software.  Unless these small increments delivered to customers are delivering fixes or are application customizations for each customer, it is unnecessary, expensive, and unworkable to release so frequently for most customers.  Further some features require long lead times to assemble, making them impossible to deliver in a short interval.   It’s like saying no matter what size building you give a team to build, they can build it in a month.

Intense Management Oversight

Traditional methodologies have strong management oversight.   Sometimes there are too many managers in these organizations, too many cooks spoiling the broth.  However, strong management practices benefit projects.  Teams need leadership to succeed.  The Agile proponents eschew strong management favoring self-organizing teams and trust.   People rarely self-organize into a productive team.  If you want an example of that, just look at the divorce rates in the US.  The best proof I can give is that you don’t find any models of it in actual practice.  All our successful models in business and sports are deliberately organized teams.  The successful teams that I’ve worked on had involved strong management and leadership.  Elite athletes require the aid of a coach and so do elite engineers.

I’ve always sensed a bit of an anarchistic mindset in the software culture, and the idea of trust as promoted in the manifesto evokes that feeling for me.  Of course we should trust people; most importantly we should only hire people that we believe are trustworthy. However, when trust is advocated to keep management from being informed, involved, and from exercising its authority, that’s anarchy.   Good leadership and good management oversight are essential to team and project success.  President Reagan put it well when he said, “trust but verify.” 

Conclusion

Many of the Agile practices aren’t new.   I’ve seen many of them in practice throughout my career; they’ve been the contributing factors in low quality, high costs, budget overruns, low team morale, and failure to satisfy customer needs and expectations.  These practices are naïve and at best suited for the smallest of projects.   A lack of discipline is the enemy of project success, and the Agile proponents have packaged an undisciplined model and have attempted to make a virtue of it.  I fully empathize with many of the challenges and failings of traditional development methodologies.  We need less religious zealotry in this practice of software development (if you have any doubt about the zealotry, just look at the words we use to describe things: extreme, evangelist, guru, etc.), choosing only to be open to a practice if it fits ones chosen creed. 

However there are wrong models that seem to explain things.  Ptolemy, a second century astronomer, best exemplified that with his elaborate model for describing the movement of celestial bodies.  No matter how many cycles and epicycles he added to his model, it would never completely describe the movements in the solar system.  On paper it looked good; it described all the observable behaviors until a new one was discovered that the model could not predict.  In response, Ptolemy identified a new cycle that would explain the motion.  It was only when it was accepted that the sun was at the center of our solar system could the motions of our solar system be unfailingly modeled and explained.   Similarly, the Agile practitioner may recommend more frequent iterations of shorter duration to remedy software management problems. While in software we don’t require revolutionary thinking to advance our art, we do need to advance the right model, and I’m afraid that Agile methodologies don’t fit that bill.

Further Reading

• Integrating Agile Development in the Real World (Programming Series) (Programming Series)
• Agile Management for Software Engineering: Applying the Theory of Constraints for Business Results
• Agile Project Management: Creating Innovative Products (The Agile Software Development Series)
• Agile Project Management with Scrum (Microsoft Professional)
• Lean Software Development: An Agile Toolkit for Software Development Managers
• Extreme Programming Explained: Embrace Change (2nd Edition) (The XP Series)
• Extreme Programming Refactored: The Case Against XP
• Pair Programming Illuminated
• Extreme Programming and Agile Processes in Software Engineering: 5th International Conference, XP 2004, Garmisch-Partenkirchen, Germany, June 6-10, 2004, … (Lecture Notes in Computer Science)
• Managing the Development of Large Software Systems