Agile Isn’t a Process
I’m reminded of an interview I had with a young company for a Software Director role. They were at point where they were ready to begin rapidly acquiring new customers. When a new customer signed, there was customization required to have the system operate according to the customer’s demanding requirements. The turnaround time from signing the new customer and having them productively using the system was approximately four weeks - at least that was the goal.
The CEO was looking for someone experienced with delivering short release cycles. His thinking was that a short release cycle had unique characteristics that are not present in long release cycle projects of six to nine month durations. He identified as one characteristic that there is no room for error in a short release cycle because there is no time to make it up. With a longer release cycles, of six to nine months, there is slack in the timeline that could accommodate things going wrong without compromising the release date. Obviously this CEO never worked on long release cycle projects where many are behind schedule before they start.
A four week release cycle seems fit for an Agile approach. Having short release cycles is one of the Principles behind the Agile Manifesto. Agile favors an iterative approach to software development, but this release doesn’t require any iterations. Therefore, it’s not clear to me that Agile is fit for this project since the Agile methodology relies upon the feedback over a series of iterations to arrive at the required solution. An Agile Methodist might say, “Well, that is easy to solve. Have a delivery once a week for a total of four iterations.” Perhaps that would work, but is it enough?
It’s Not The Process
Let’s assume for a moment this problem was one designed specifically for a process methodology like Agile to solve. Is that all there is to consider? Assuming a team could deliver successfully to the four week timeline will this be sufficient to address the demands of this growing company? If the company is signing customers at an accelerating rate, it won’t be long before the backlog of customers grows to unmanageable levels. The company can hire more people to support the growing demand, but it takes time to hire and train people; therefore, while the team is growing and ramping up to speed, the backlog of work continues to grow.
Looking to the software development process to solve the delivery requirements of this growing company is a mistaken solution to this company’s aggressive demands, but it’s a solution too often proposed by experienced professionals in the business. To be Agile in this environment, and all environments that I’ve worked in, requires entirely removing the bottleneck in the software development organization. The software department needs to find solutions that require less programming, less testing, and less IT personnel - and even none if possible.
Let’s consider some business examples to illustrate the point. What if the IT department needs to be involved to add a new book, to add a new customer, or to change an item price on Amazon.com? Would the software development process be fast enough for Amazon.com to remain competitive? Or let’s take a web hosting company, what if the IT department needed to get involved for each new customer that signs up? Would the software development process be sufficient for any web hosting company to remain competitive? What if every blogger using Worpress needed to go to Wordpress.com for custom site designs? Would the software development process be sufficient for Wordpress to remain competitive? Could Wordpress ever adequately meet its customer’s myriad and demanding needs if the skin was not customizable by the end user? Would Wordpress have as large a market share as it has now if the software was not end user customizable?
To be more agile, the IT department needs to rethink its role in the organization. The role isn’t to be the high priests of some complicated methodology or technology, but their role is to identify the technologies and methodologies that permit the company to compete in the marketplace more efficiently and strategically while reducing costs and time to market. It’s not to apply the hammer to every problem, but to create the tools, technologies and/or processes that are appropriate for each and every unique business requirement.
Where To Focus
While emphasis on software development methodology will never go away and shouldn’t, the software process methodology is not where the IT department can significantly reduce costs and time to market, and increase agility. Every study has confirmed that effort and software size are directionally correlative: the more code that is written, the more effort that is required to complete the delivery. Assuming we have a problem where the best solution requires one million lines of code to complete and all the requirements are needed to be delivered day one to have a marketable product, what option will reduce costs and time to market, and increase agility the most? Will it be choosing Agile over Waterfall or will it be finding other ways to solve this problem? For all but the most naïve, finding another way to solve this problem will yield the largest returns.
All the significant gains in software development productivity have come from improvements in technology. Let’s examine some of them.
- The introduction of high level languages significantly improved programmer’s productivity where one high level language statement translates into many lines of assembly language. Assuming that programmer productivity is 10 lines of code per day, writing 10 lines of high level language code will yield more functionality than 10 lines of assembly language code, thus the productivity of functionality delivered has increased. In addition, the improved readability of high level languages improved the ability to understand the implementation details faster. A quicker learning curve makes the software easier to maintain.
- Object oriented languages significantly improved productivity over procedural languages such as C or Pascal. With an object-oriented language you can reuse code easily. Templates and inheritance provide for reuse in significantly improved ways. Solutions are better abstracted: they more closely model the problem domain.
- Standard libraries, design patterns, and standard protocols significantly improved productivity for a number of reasons. One, there is no need to reinvent the wheel reducing the effort in design and implementation. Two, the solutions have been tested and confirmed to be appropriate by other experiences. Three, there is a rich supply of experienced professionals with knowledge of the technologies to draw upon.
- Software development tools, such as defect tracking software, integrated development environments, software debuggers, configuration management software, compiler technology, and even syntax aware editors to name a few, have all improved the software developer’s and the team’s ability to build software faster and more reliably.
How much longer do you think a release cycle would be if we didn’t have these technological advances? What if we were still coding our software in assembly language? What if we were still managing our defects in loose leaf binders as I did early in my career? What if we didn’t have the advantages of feature rich libraries and foundation classes? If we used Agile to solve our problems using the old technologies, would we see as much improvement as we have with these technology advances?
Being Agile
Like the productivity gains provided to software development teams by the advances in software development tools, we can improve our productivity with similar advances to our business problems. We need to understand the essence of the customer need and relinquish control to the customer, customer teams, and/or business partners. Once we understand the customer’s changing needs, we can architect the solution in a way that supports making those changes easier, faster, and with higher quality. This may involve one or more of the following approaches
- Provide a messaging protocol in a standard format such as XML.
- Provide an API to the functionality that a customer may want to reuse in other ways or to customize the application such as done with Wordpress.
- Provide an editing interface to the application for the features that are likely to require changes by the customer. For example, if the system is rules driven, provide the user with the ability to edit rules for themselves. Make the rules data driven, don’t hard code the rules in whatever chosen language the system is implemented.
- Provide tools that change the behavior of the system. Some of these tools can be pushed out to the end users, customer teams, and even business partners. In other cases, the tools would generate the code that needs to be built by the development team.
- Provide a rich architecture that more closely mirrors the problem domain, is easily extensible, and is orthogonal.
These are only a few of the many possibilities that are available to us. In many cases, the possibilities are unique to the domain. Some of these options need to be attenuated by the life expectancy of the product as providing these capabilities require an investment, and the return may not support the investment with a product having a short life expectancy.
Delivering architecture and tools with these capabilities requires a large investment in up front analysis and design. Developing these solutions often requires a long development lead time before the first build is available for test - it’s typically longer than a few months. What you give up on the front end, though, you gain in leaps and bounds in the back end. After all, if we are to believe the tired maxim that 80% of the effort is spent in maintenance, doesn’t it make sense to design solutions that will optimize the maintenance phase?
Conclusion
The Agile methodologies are not suited for delivering solutions of this nature. An emergent, iterative approach does not have much to offer as a solution when the requirements for solving the problem require a large up-front investment in analysis and design. The Agile methodologies are designed to solve today’s problems instead of anticipating tomorrow’s problems and instead of developing architectures and tools that anticipate them. Agile approaches favor delivering something quickly at the expense of a well thought out architecture and at the expense of anticipating future requirements.
While Agile with a Big “A” is a development process, being agile has little to do with the development process. Improving a company’s ability to be agile, to deliver quicker, and to deliver higher quality requires the IT department to take an even broader perspective. They need to look at the end-to-end business processes, develop tools and processes that leverage the corporate resources to their fullest in the interest of delivering the greatest value to the customer and to the shareholders. Optimizing the software development methodology to achieve agile goals is too narrow a perspective to solve this problem. Of course, we should improve the Software Development Processes, not so much to be agile, but to be more efficient, to be repeatable, to improve quality, and to make better decisions. Agile as a solution for being agile is a misnomer.
(No Ratings Yet)
Loading ...
[...] Upfront Design, YuWantItWhen.com trackback Head over to Bill Miller’s blog to read his great post on Agile on and applying the appropriate problem solving techniques to business processes. Bill [...]
Wow - great post! I’ve added you to my blogroll and published a post to this great analysis. I’ll shut up now, as I have nothing to add save that this is what I call great advise and true wisdom.
Dave thanks for the positive feedback. I’m glad you enjoyed it.
Really, this is serious and fair assessment that is offered that counters the zealotry and religious overtones that many of Agilista exhibit. Great, discerning analysis.
Minor nit, C and Pascal are not functional languages, they are procedural imperative languages.
Allan, thanks for the correction. I’ll fix that in the text.
Hi Bill,
Not every time. I see a place for Agile methods in “anticipating tomorrow’s problems”. To say that since Agile methods don’t make large up front investments in design, therefore Agile isn’t good for solving problems that, by assumption, require large up front investments in design isn’t saying much. But who assumed that such large investments are *always* required for certain kinds of problems? Logical necessity? No proof needed? It’s obvious? Oh?
According to the Wikipedia, the purpose behind the Agile family of software development techniques is to minimize project risk. A key weapon is Agile’s emphasis on producing working software on a short, iterative cycle. (Agile methods appreciate the difference between arguing and demonstrating a partial solution.)
Logically speaking, the amount of time or thought put into a software’s initial architecture, per se, has nothing to do with controlling the level of risk. Because, in the end, how do you tell if the software will actually perform as intended? There is only one way - Run It.
It has been my experience that taking a long time to architect software only guarantees complex software, not software that will actually work.
To do something *really* difficult/original, as opposed to merely complex, requires a simple (not necessarily easy) basic architecture at the core. Often something that seems too simple, at first, to be workable. No need to spend much time on it. Then over time knowledge is gained from working software and a full solution is implemented. I’ve used this technique with great success.
George,
Thanks for reading and sharing your point of view.
I don’t agree a proper up front investment implies a complex solution, and I don’t agree that risk is reduced because of a shorter interval.
I do make the point in the essay that for some problems it doesn’t make sense to have a heavy up front investment. I advocate the up front investment to be as long or as short as required to do the proper job within other necessary constraints.
A short investment in up front analysis doesn’t guarantee a simple design anymore than a large investment in up front design guarantees a simple design. Simple designs happen when we have a good understanding of the problem, and we have the right people working on them. Smart, experienced people develop simple solutions and they arrive at them quicker than most. Having a good understanding of the problem usually requires an investment in time to understand it well, and the time required to understand the problem well increases as the problem to solve is bigger and/or more difficult.
My reference to large up front investment is to contrast it with the inadequately small up front investment advocated by Agile proponents. I don’t believe we can do a proper job by time boxing design or artificially setting a week or month iteration to deliver a feature set. I’m all for a week iteration if it’s adequate time to solve the problem properly.
Our experiences seem to be very different. The overly complex and bug ridden software that I’ve seen has been the result of a solution developed quickly to get something out the door quickly and then fixed quickly to keep it going. As future releases are built on top of the original architecture, the software further decays and becomes more complex as solutions are further coded into a fragile architecture because there is no time in the schedule to rework the original solution, so the developers have no choice but to solve new problems in more complex ways as new requirements are levied on the system. Usually in these environments there is no time to rework what could have been avoided with some more up front design and analysis — enough to be successful, no more, no less.
Great people investing the proper amount of time in design and analysis produce the best solutions.
I do accept your position that we should be striving for simple solutions, but it often takes time to arrive at a simple solution, often more time than to arrive at a complex solution.
Hi Bill,
I also agree than every problem is potentially unique. But yes, it seems we have different default approaches to software design. As you mentioned, perhaps it’s because of different experiences. So I am *not* saying one is better than another.
But it might be interesting to try and sharpen these distinctions.
Let me reiterate that I don’t consider most business software to be very difficult — rather it tends to be complex. Business software is some of the most complex software there is. Say millions of lines of custom Java code on top of some proprietary framework on top of the J2EE. Business software is much more complex, but far less difficult, than say NASA’s trajectory software for getting our latest Mercury probe (MESSENGER) into orbit around that tiny planet (by using a series of gravity assists from Earth, Venus, and Mercury itself). Easy to visualize the solution, hard to actually do.
Now, our workhorse method for managing complex problems is divide-and-conqueror. We turn a big complex problem into a bunch of smaller, simpler problems. Agile methods naturally tend to enforce this approach. Big up-front analyses do not naturally enforce this approach. (Although if the big up-front analysis is also object-oriented…that’s a different discussion.)
So I tend to favor an Agile approach by default. And thus, I tend to favor working code over design documentation. I would usually rather build upon a bunch of working code pieces than implement all new code from a design specification.
Second, I trust working code more than verified design documents. Software quality is very difficult to obtain. There is significant risk in even verified and validated designs. As they say in science: Theory is fine, but the sole test of knowledge is experiment. There is no substitute for working code. And again, this is something Agile methods tend to naturally enforce, but that big up-front analyses do not.
Hi George,
I appreciate your interest in expressing your thoughts. This is exactly the kind of exchange I was hoping to have when I started this blog.
There are a few points you make in your response that I have a reaction to, but I’d like to focus on the one point you make about Agile practices enforcing the approach of breaking bigger problems into smaller problems.
I studied CS in 1981 when I entered college as a Freshmen. Decomposing a problem into managable components were practices we studied in college way before the Agile manifesto was birthed. On my first job in 1985, I programmed in assembly language, PL/C and Pascal. Objected oriented programming was still an idea in academia back then. During those years we did big up-front designs, but we functionally decomposed them and assigned smaller functional components out to team members to develop them in more detail.
Good teams have been decomposing problems into smaller managable components for many years. So I wonder why do the Agile proponents insist they’ve developed something new when us old timers have being doing this stuff for decades, and why do they feel they’ve created new laws of software development when these practices are anything but new? These are not concepts brought to the table by Agile; Sorry, there’s nothing inherent in doing big up-front design and analysis that prohibits or deters functional decomposition. In fact, it’s embraced by traditional practitioners, and it’s been embraced for decades. Functional decomposition is simply a good problem solving technique no matter the discipline or production methodology.
Really this whole Agile movement feels like some twilight zone episode where some kid in a remote village sees a boulder rolling down a mountain and gets the idea for a wheel, and he thinks he discovered something new.
Hi Bill,
Feel free to take the discussion in any direction you think might be most interesting. It’s your blog, and I certainly don’t think I have any great insights to share
I don’t think I said, and I certainly did not mean to imply, that big up-front designs normally don’t do some kind of problem decomposition — functional, structural, object, etc. Rather, my point is that with Agile methods, because of their short iteration cycles, you simply don’t have much choice. And that’s good.
And my bigger point, again, was that since the Agile goal is working code each iteration, the decompositions must actually be real solutions. That’s even better.
BTW, I think we need a better term than “big up-front design”. Almost sounds like we’re belittling it. Maybe a “conventional design phase” is better?
Actually, my biggest problem with agile methods is their weakness in assisting in using the other main technique for attacking complex problems — analogies or paradigm shifts. I believe it is easier to get a high level view of a complex problem using a conventional design phase. And that makes coming up with a useful analogy or paradigm easier. Agile methods are way too fast and too low level for this in my opinion. (But someone could change my mind with some evidence.)
I guess I should explain paradigm shifts a bit. There three basic types: natural, artificial, and what I call synthetic.
A natural paradigm is encountered a lot in object oriented programming. We can have a logical car object, checkbook object, or screen object. By using such real-life analogies it makes our complex problems easier to understand and work with. We already understand these natural things.
I was able to implement a robot/facility message handling system at an automated factory once (this was just before the internet) by simulating the US postal system. Electronic messages became letters, junk mail, priority mail, etc. Every robot had its logical mail box complete with flag. Some computers turned into post offices. Servers became mail centers. Every message had its zip codes. Etc. We knew it would work — the mail does get delivered. And we could even explain what we we doing to managers
An artificial paradigm is where the problem is expressed, at least in part, in terms of abstract logical and mathematical entities. Perhaps the classic example of an artificial paradigm shift is mapping data from a hierarchical file structure to a relational database.
A synthetic paradigm is where you take a common natural or artificial paradigm and modify it. A good example is a spreadsheet. Naturally, it represents a sheet of paper with rows and columns where you can write numbers. But then we add the ability to put live formulas in the cells. Something we can’t do in nature but a snap in the abstract computer world, and it changes everything about what spreadsheets can do.
Hi George,
I agree. BDUF does have negative connotations for many. Though, I do believe it fairly characterizes the approach when contrasted with the Agile approach, which is to just-jump-right-in-and-start-coding. Compared to that approach everything looks big.
I’m unconvinced there’s a correlation between a short cycle and a properly architected solution, nor do I believe it’s a function of the development methodology forcing a simple solution. It’s a function of the quality of the management and the team working the solution. Good people given sufficient time will architect good solutions. Less talented people given any amount of time (lots or little) will architect poor solutions.
When you describe how a conventional design phase supports creating a proper paradigm, it is exactly the point I am trying to make in the article, and finally, it’s essence of agility. When teams are building solutions on properly architected products, they are more agile. They can deliver faster and they can respond to change quicker. That’s what this essay is all about.
Well said Bill.
Thanks for the essay. I enjoy reading your blog.
George
Thanks George for taking the time to read the postings and most importantly for sharing your views. I do appreciate it.
I do like your explanation of the different paradigm shifts. I’ve never heard it explained that way before, but it makes a lot of sense, and I like the terminology. I will have to remember it.
Before I comment I do have to caveat — I didn’t have enough time that reading your text deserves, to fully digest & savor it. It seems very dense, and I look forward to reading it in-depth.
But I did want to point out that you seem to have fallen into the (common) mis-belief that scrum = agile. Scrum can be performed as an agile practice, but so can a waterfall SDLC. Agile is not a predefined set of tools and processes but rather a set of meta-directions about how to use tools & processes.
I’ve seen scrums that weren’t Agile, and I’ve seen waterfall projects that were. In your text you say “Agile methodology relies upon the feedback over a series of iterations to arrive at the required solution. ” No, it doesn’t.
The Agile principles (that pertain) are
* Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
* Business people and developers must work together daily throughout the project.
Scrum, when performed in an Agile manner does implement the Agile principles, but your statement implies that an “Agile methodology” requires a series of iterations, doesn’t seem to be entirely correct. However, there is some wiggle room here, based on semantic differences, which I acknowledge.
Thanks
Terry
Terry, thanks for visiting and offering your views. Here’s one of the emerging problems I’m seeing with Agile. As I write more about Agile and I talk to more people about Agile, I find that all the Agile proponents have a different and often conflicting view of what it means to be Agile. I’m not surprised as the Manifesto leaves too much to the imagination. We need verions 2.0 of the Manifesto to clarify it’s intent, so we all know when we see Agile in practice, the practice meets the Agile standard. Until then, Agile will continue to be hijacked by those who hate process to defend what they are doing as Agile. And for the rest of us trying to figure out whether a practice is really Agile or not, it’ll continue to be like playing the game To Tell The Truth: Will the real Agile please stand up.
Leave a comment!
EMAIL
Subscribe
Visitors
Polls
Categories
Blogroll
Tag Cloud
Agile Analysis BDUF Career Certifications CMMI Critique Debate Defect Free Design Estimating Ethics excellence Forum Future good enough Hiring Leadership LOC Management Methodology Metrics Outsourcing Popular Process Product Management Project Project Management Project Tracking Quality Recruiting Refactoring Requirements Schedule Scrum Skills Stable Test Strategy Tracking Traditional Upfront Video Waterfall wrap-up XPRandom Posts
Latest Video Post
Recent Comments
Most Commented
Most Viewed