You Want IT When? » Danger: Agile Practices at Work

Sat, 12/09/09 – 9:21 | No Comment

The common thread for all human pursuits is our nature. This is best exemplified in our current economic crisis and the events leading up to the dénouement. Peter Schiff is a market participant and analyst who was scorned and laughed at for his prescient conclusions on the US financial markets and economy.

Read the full story »

Management

Methodology

Metrics

Quality

Requirements

Home » Methodology

Danger: Agile Practices at Work

Submitted by Bill Miller on Tuesday, 28 August 2007 9 Comments

Bataan Death March

There’s a lot of activity and promotion of the Agile software methodologies that can be found on both the Internet and in books today. Its adherents are zealous in their commitment to the Agile principles codified in the principles behind the Agile Manifesto. I first became interested in the Agile development practices when the Extreme Programming practices began generating a lot of interest in the software community. My first reaction to this was to be turned off. The name conjured up images of chaos, cowboy programmers, and death march development projects — practices I am all too familiar with. I found the advocacy of pair programming to be unnatural and impractical.

After reading many articles published by Agile proponents, I remain steadfast in my beliefs that something is wrong, and rather than advancing the state of the art practices in Software Development, Agile proponents are setting us back, and in this world where jobs easily cross international boarders, especially easy in software, we need practices that demonstrate the value proposition for keeping teams here in the US: practices that deliver reliably and predictably on commitments and practices that can demonstrate improvements in productivity and quality.

Wikipedia explains, the Agile practices developed out of a reaction against “heavyweight” methods. The Agile community has some valid gripes with traditional practices. Traditional practices have had their share of problems, but are the solutions advocated by the Agile community a step in the right direction? I believe not, and I’d like to compare and contrast some of the problem practices of traditional methodologies against the practices advocated by Agile methodologies.

Overboard With Documentation

Traditional practices often go overboard in requiring documentation, often producing many pages no one ever reads or uses. That needs to be corrected. The Agile proponents promote oral communication over documentation. Oral communication is vital. There is nothing like the fidelity that can be had in a good conversation and it saves time. However, there is tremendous value in formal communication – especially having written requirements. Documentation frees people’s minds for new information. Memories often fail and people never remember every detail quite like it was intended. Put two or more people in a room to agree on requirements, and each will remember some of the details of the requirements differently. Further, formal requirements allow other people to review the details and provide further input. Reviews are a best practice when done intelligently. There is no opportunity for review without a written record.

Resist Change

Traditional practices often resist change — especially major requirement changes after the requirements have been signed off. The practice sometimes leaves the business group feeling that the software team doesn’t share the same goals as the business. The software team gets a reputation for delivering less than what was expected. That’s never positive for the software team. The Agile proponents promote and value continuous change, anytime in the life cycle. But too much change is inefficient, expensive, and counter productive to progress and team morale. I’d love to know what applications Agile practitioners are building because in my 20+ year’s experience, requirements rarely changed significantly on my projects. Instead, they grew. When requirements are changing too much and too broadly, it’s a signature of a project that is ill conceived and lacks good leadership. This is nothing new, and it has been the cause of more project failures since the beginning of software development.

Rigid Project Life-cycle

Traditional practices have promoted a sequential phase strategy often requiring one phase to complete in its entirety before beginning the next phase. This is a very rigid, inefficient, and tedious approach to software development. The Agile proponents have promoted more of a parallel delivery strategy. Some proponents of Agile methodologies seem to deny the existence of dependencies and the existence of a natural waterfall approach to software development. The reality is that there is a natural waterfall behavior to the software development life-cycle: there is no avoiding the natural progression of requirements, design, code, and test phases of software development. You can’t test what hasn’t been built, and you can’t code what hasn’t been defined. While some traditional practitioners have followed a rigid waterfall approach, most traditional practitioners maximize parallelism. It is difficult not to when the time pressures are omnipresent.

Big Bang Approach

Traditional methodologies have promoted a big bang integration and test strategy. While it is commonly accepted that big bang integration approaches have major drawbacks, it is likely that teams will continue to engage in this approach, and in some very big projects it may be unavoidable. The Agile proponents have promoted continuous test and short incremental deliveries. Incremental deliveries are a best practice that traditionalists embrace. Where the Agile community has it wrong is that the increments are normally time-boxed. Many deliveries require longer durations than the short time intervals advocated by Agile practitioners. Traditional practitioners let the delivery dictate the interval as it can only be accomplished.

Upfront Definition

Traditional methodologies have relied upon considerable up front definition and design. When done well, the applications are built upon solid architectures that accelerate future deliveries. Agile methodologies promote an evolutionary approach to definition and architecture believing it’s not possible to identify accurately up front enough of the requirements and architecture for constantly changing customer requirements. The mantra in the Agile community is constantly refactor. This is a highly expensive approach, and in this age of globalization where costs drive decisions, it should be looked at with trepidation. This isn’t a wise approach, as market pressures to deliver quickly always constrain your latitude for refactoring. If it’s a choice between delivering something two months late to improve architecture and delivering early with the old architecture, most times delivering early wins at the expense of further architecture decay. Constant refactoring is expensive. You’ll often hear teams in the trenches lament about how it’s too expensive to do it right the first time but it’s never too expensive to do it over, and in the case of Agile, over, and over, and over, and over…

Infrequent Large Deliveries

Traditional methodologies tend to deliver new functionality to customers less frequently, and they consequently have the reputation of being unresponsive to customers. Agile proponents promote frequent releases from “a couple of weeks to a couple of months.” Most customers will not adopt such rapid changes. It costs time and money to deploy and train people on any new software. Unless these small increments delivered to customers are delivering fixes or are application customizations for each customer, it is unnecessary, expensive, and unworkable to release so frequently for most customers. Further some features require long lead times to assemble, making them impossible to deliver in a short interval. It’s like saying no matter what size building you give a team to build, they can build it in a month.

Intense Management Oversight

Traditional methodologies have strong management oversight. Sometimes there are too many managers in these organizations, too many cooks spoiling the broth. However, strong management practices benefit projects. Teams need leadership to succeed. The Agile proponents eschew strong management favoring self-organizing teams and trust. People rarely self-organize into a productive team. If you want an example of that, just look at the divorce rates in the US. The best proof I can give is that you don’t find any models of it in actual practice. All our successful models in business and sports are deliberately organized teams. The successful teams that I’ve worked on had involved strong management and leadership. Elite athletes require the aid of a coach and so do elite engineers.

I’ve always sensed a bit of an anarchistic mindset in the software culture, and the idea of trust as promoted in the manifesto evokes that feeling for me. Of course we should trust people; most importantly we should only hire people that we believe are trustworthy. However, when trust is advocated to keep management from being informed, involved, and from exercising its authority, that’s anarchy. Good leadership and good management oversight are essential to team and project success. President Reagan put it well when he said, “trust but verify.”

Conclusion

Many of the Agile practices aren’t new. I’ve seen many of them in practice throughout my career; they’ve been the contributing factors in low quality, high costs, budget overruns, low team morale, and failure to satisfy customer needs and expectations. These practices are naïve and at best suited for the smallest of projects. A lack of discipline is the enemy of project success, and the Agile proponents have packaged an undisciplined model and have attempted to make a virtue of it. I fully empathize with many of the challenges and failings of traditional development methodologies. We need less religious zealotry in this practice of software development (if you have any doubt about the zealotry, just look at the words we use to describe things: extreme, evangelist, guru, etc.), choosing only to be open to a practice if it fits ones chosen creed.

However there are wrong models that seem to explain things. Ptolemy, a second century astronomer, best exemplified that with his elaborate model for describing the movement of celestial bodies. No matter how many cycles and epicycles he added to his model, it would never completely describe the movements in the solar system. On paper it looked good; it described all the observable behaviors until a new one was discovered that the model could not predict. In response, Ptolemy identified a new cycle that would explain the motion. It was only when it was accepted that the sun was at the center of our solar system could the motions of our solar system be unfailingly modeled and explained. Similarly, the Agile practitioner may recommend more frequent iterations of shorter duration to remedy software management problems. While in software we don’t require revolutionary thinking to advance our art, we do need to advance the right model, and I’m afraid that Agile methodologies don’t fit that bill.

But I remember reading the road less traveled, and he made a point that resonated with me. He said life becomes easy when we submit to the struggle. Similarly, software becomes easy when we don’t wish it to be so easy. Submit to the struggle, and your projects and your health will be better for it.

- 3 November 2008 at 11:25 pm

Pargev said:

First of all, thanks for the interesting point of view. I am not an Agilist, and can’t say that Agile is the best solution to all situations. There are cases when you need to implement a mature process with strict waterfall workflow. However, please find my comments per each section of the article, criticizing the critics. My vision is that every methodology may be learned and tools and techniques should be studied.

Overboard With Documentation:
Project Management (Agile as well) without documentation is a chaos. Agile practitioners should have the documentation of the requirements, having them informal and shortcutting the definitions of the text fields by leaving this kind of details to the common sense and discussion with the developer is the way to practice a good agile approach which saves time indeed. Without a user-story documented, process should not start. No documentation = no project success. I think this point is exaggerated.

Resist Change:
Customers often change their minds in the meantime of the project running. To fit into new business goals Agile says that it accept change. I will argue that Agile SHOULD NOT accept any change in the middle of running iteration, but should plan that change to the next iteration instead. With iterations keeping short, the quick reaction to the changed business need may be achieved. Almost all projects in all companies I’ve worked in were having significant scope changes.

Rigid Project Life-cycle:
Consider TDD, it really works the way the test suite (unit and automated system tests) is designed upfront the development code.

Upfront Definition:
Agree, most of the Agile practitioners think a good design is not a mandatory for long term projects. Well, it is.

Infrequent Large Deliveries:
For the products which require much time and effort for deployment, consider having the QA environment (which is by all means mandatory to have, I think)shared with the customer for the demo of the current state. Frequent feedbacks from the customer is very important whether that deliverable is a deployment to the customer environment or a web-meeting demo.

Intense Management Oversight:
The project manager is not a project manager if he is not being informed and involved. The agile does not say to do so. The rate of the marriages is more than the rate of divorces, obviously. The PM should recognize the best teams to work with each other, should keep the hand on the pulse of the project. The style of management is up to the personality of the PM, the same style fails and wins in different environments. Agile never say “do not verify”.

Conclusion:
I have my own methodology which is a mix of the all what I’ve studied and consider fitting the company’s and my needs as a PM. I would welcome a discussion.

- 8 April 2010 at 1:10 am

Bill Miller (author) said:

Pargev, thanks for reading and taking the time to comment. I wrote this a while back, but I still like a lot of it after reading it again. There’s no perfect way to build a software team and execute a project.

But what we do know is that there are some practices that result in better results than others. For example, up front design USUALLY results in a better architecture, higher quality, and less cost over the lifetime of the project. Of course, not every problem requires an investment in up front design. But to position that up front design has no benefits and should not be practiced as agile proponents often do, does a disservice to their process and the profession.

Or verbal communication over written documents. Sure I hate writing some of these documents. Some are even useless, but producing the right documentation and doing a great job of it improves quality and results. For example, if you’re writing an API and their are other team members who will be a consumer of that API, having it well documented up-front allows everyone to work independently and in parallel. Also, the benefits of having a good up front review of the API before it’s committed to code saves time and improves the quality. I know firsthand because I’ve benefited from the feedback of more experienced team members when I was tasked with writing an API on projects in the past. It saved time, improved the quality, and it allowed the API consumers to work independently and in parallel before the API was available to link into a build.

Where I’m at odds with Agile is when it does not acknowledge that many of these practices have merit, and there are more reasons to do them than not do them.

Finally, every instance of a process should fit the goals, the domain, and nature of the team, but even then, it should never be a reason to do the wrong things or leave out the right practices.

- 8 April 2010 at 8:29 pm

Pargev said:

Dear Bill,

Agree to all your points. There is no ideal methodology to be used in all projects/companies. And I disagree with the agile proponents who say that the documentation is evil and up front design is redundancy.

To me, Agile is a mindset of work and some tools which may be applied to increase the visibility for the customer, improve the communications in the team, raise issues as early as possible, etc. What pure agile methodologies do not cover, IMO, is the quality and risk management, business level performance reports, as well as many other useful practices which may be not that pleasant for the developers and testers, but are must-have for mid-high management in mid-large companies.

I think I am on the same page with you.

Thanks for the interesting viewpoints, again.
Pargev

- 10 April 2010 at 7:59 am

Bill Miller (author) said:

Pragev,

I believe you nailed one of my themes in this blog when you say, “useful practices which may be not be that pleasant for the developers and testers, but are must-have for mid-high management in mid-large companies.”

Project success comes when we embrace those things that we find hard and don’t like to do but must if we are to be successful. It seems as the Agile solution to those problems is to argue why they are unnecessary rather than propose solutions for doing them better and having greater success with them.

Regards,
Bill

- 11 April 2010 at 11:14 am

Edo Civic said:

Dear Bill and visitors,

First of all, sorry for my bad english.

I’ve read some articles on this blog and I can only say “I like”. However, I decided to make a comment on this article and I hope you will take some time to read it carefully and reply to me.

Well, the main question is “What is the real Agile?”. I’ve read The Agile Manifesto more than twice and I can only underline your statements about it.

The fact is:

Companies have to change their business strategies, adopt new products, etc. very fast if they want to survive on the market. The lifecycle of the modern products are mostly very short, services are also changing very often. On top, modern companies change also their organisation structures more often facing new markets and demands.

The software system, which automates business processes shouldn’t be the bottleneck in the whole business system of the company – but it is very often.

The customer requires a change in the software system, a modification of an existing feature (change, upgrade) or a completely new feature.

If we have done everything right before, we can realize such tasks in the shortest possible time and with minimal efforts – regularly we don’t change the core of the system, only the features (well, of course in the early phase the core changes more often till the analysts and the designer have made their final decisions). That implicitly means we must have been planning possible changes durring the design of the system.

The Agile Manifesto says “Responding to change over following a plan”.

First, I thought this must be a joke. But than I saw the sentence somewhere from an Agilist “If you plan and do what you planed, you’re gonna get right that in the end, but you won’t get what you really need.”

My comment to that is very simple – “Plan what you gonna really need, but you have to know first what you gonna need.” And thats the trick imho. The most people will say “You can’t know what you gonna need! Are you magician ?” And thats the point where the story begins.

Designing software systems in modern times requires to handle its changes as a default requirement (concern), just like logging or security. This is definetely something you can plan! But, building such systems requires a larger investments of time to design and implement them. The second problem is you need very good developers which can handle such high grade of complexity (ASoC, POP experienced).

Agile suggests learning by doing imho (step by step architecture design). The first problem is on each step you have to do the whole waterfall job again and again to avoid total chaos. But, with the time, the system grows and its becoming harder and harder to manage all tasks in an iteration. And the time is ticking away … sooo .. the first thing we don’t need is the documentation, aaaand the second thing .. tests, they take anyway more time then the implementation itself. Well.. sounds like a highway to the hell – and yes it is.

The second problem is collective code ownership / weak leadership. Depending on available characters it can run good or bad, but in fact, noone of them is a really good developer so regularly you can expect a lot of trouble in the team (undisciplined hacks, Quick And Dirty soultions, functionallity replications, etc.).

The third problem is, there is not really enough time to think about the next step in the proper manner and required depth. Its some kind of McDonalds of Software Development.

So imho, the conclusion can only be: Agile fits if you need to build a small system with a couple of average developers, very fast and cost effective. If doing right, you can develop and maintain features agile with small independent teams filled with Agilists – but don’t ever let any of them design or let them influent the architecture of the whole software system. The best way is to ignore them since there cannot be a serious discussion about agile development of larger Software Systems designed for change.

Therefore my conclusion would be: real agile would define a Methodology to build a Software System, which once built has a flexibility parameter which is defined as the time needed to make the changes in the System to fit new / changed customers requirements ?

Since such systems can already be built with traditional methodologies (I think its enough to recognize future changes and change management as a concern) I don’t see the need to define something new. Agile as it is, is also ok for small (sub)systems (features/modules of the system) and it helps finding cheap developers on the market who will do the job.

The most of the problems described also in this article are also my experiences with Agile, but I don’t blame Agile for that, but I do blame people who decided to build Systems too large for Agile with Agile Methodology.

Best Regards

- 5 May 2010 at 4:49 pm

Bill Miller (author) said:

Dear Edo,

Thanks for reading my stuff and your thoughtful reply. I believe we are of similar minds on this subject. All product development is challenging.

I have many analogies for the agile vs traditional approaches. My newest one is to contrast it with investing. Some people are traders, constantly changing their portfolio positions and components based on the daily news and market fluctuations. Some of these people are extremely successful, but most aren’t. In fact most are extremely unsuccessful with this approach.

Others don’t worry much about the short term and invest for the long hall. Their portfolios don’t change much, and they sometimes under perform the market in the short term, maybe even long term. Over the long term, though, this investor is normally the most successful. Most do not go bust with the long term approach.

I believe it’s the same for product development. Long term investment in your products have the highest likelihood of success. It took 3 versions of Windows over many years before Microsoft Windows dominated the market, and guess what? They still do. Most don’t have the confidence or the courage to invest properly in the long term, so they tilt at windmills. It’s soothing.

Best Regards,

- 5 May 2010 at 6:45 pm

Edo Civic said:

Hello again,

right there is my problem. Some of our customers were pretty dumb and had a ride on the Agile hype and after losing the most of their money and trust in such companies they come to us. So we are facing the phenomena that we do the work right and bring the business of our customers running, but the others take the big money !? If the trend continues we will not be able to survive.

Agile seems to be the dominant Methodology at the moment(I guess because of the large number of web development projects) and that makes me really scarry.

Best Regards

- 5 May 2010 at 8:07 pm