But I did want to point out that you seem to have fallen into the (common) mis-belief that scrum = agile. Scrum can be performed as an agile practice, but so can a waterfall SDLC. Agile is not a predefined set of tools and processes but rather a set of meta-directions about how to use tools & processes.

I’ve seen scrums that weren’t Agile, and I’ve seen waterfall projects that were. In your text you say “Agile methodology relies upon the feedback over a series of iterations to arrive at the required solution. ” No, it doesn’t.

The Agile principles (that pertain) are
* Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
* Business people and developers must work together daily throughout the project.

Scrum, when performed in an Agile manner does implement the Agile principles, but your statement implies that an “Agile methodology” requires a series of iterations, doesn’t seem to be entirely correct. However, there is some wiggle room here, based on semantic differences, which I acknowledge.

Thanks

Terry

I do like your explanation of the different paradigm shifts. I’ve never heard it explained that way before, but it makes a lot of sense, and I like the terminology. I will have to remember it.

Thanks for the essay. I enjoy reading your blog.

George

I agree. BDUF does have negative connotations for many. Though, I do believe it fairly characterizes the approach when contrasted with the Agile approach, which is to just-jump-right-in-and-start-coding. Compared to that approach everything looks big.

I’m unconvinced there’s a correlation between a short cycle and a properly architected solution, nor do I believe it’s a function of the development methodology forcing a simple solution. It’s a function of the quality of the management and the team working the solution. Good people given sufficient time will architect good solutions. Less talented people given any amount of time (lots or little) will architect poor solutions.

When you describe how a conventional design phase supports creating a proper paradigm, it is exactly the point I am trying to make in the article, and finally, it’s essence of agility. When teams are building solutions on properly architected products, they are more agile. They can deliver faster and they can respond to change quicker. That’s what this essay is all about.

Feel free to take the discussion in any direction you think might be most interesting. It’s your blog, and I certainly don’t think I have any great insights to share

I don’t think I said, and I certainly did not mean to imply, that big up-front designs normally don’t do some kind of problem decomposition — functional, structural, object, etc. Rather, my point is that with Agile methods, because of their short iteration cycles, you simply don’t have much choice. And that’s good.

And my bigger point, again, was that since the Agile goal is working code each iteration, the decompositions must actually be real solutions. That’s even better.

BTW, I think we need a better term than “big up-front design”. Almost sounds like we’re belittling it. Maybe a “conventional design phase” is better?

Actually, my biggest problem with agile methods is their weakness in assisting in using the other main technique for attacking complex problems — analogies or paradigm shifts. I believe it is easier to get a high level view of a complex problem using a conventional design phase. And that makes coming up with a useful analogy or paradigm easier. Agile methods are way too fast and too low level for this in my opinion. (But someone could change my mind with some evidence.)

I guess I should explain paradigm shifts a bit. There three basic types: natural, artificial, and what I call synthetic.

A natural paradigm is encountered a lot in object oriented programming. We can have a logical car object, checkbook object, or screen object. By using such real-life analogies it makes our complex problems easier to understand and work with. We already understand these natural things.

I was able to implement a robot/facility message handling system at an automated factory once (this was just before the internet) by simulating the US postal system. Electronic messages became letters, junk mail, priority mail, etc. Every robot had its logical mail box complete with flag. Some computers turned into post offices. Servers became mail centers. Every message had its zip codes. Etc. We knew it would work — the mail does get delivered. And we could even explain what we we doing to managers

An artificial paradigm is where the problem is expressed, at least in part, in terms of abstract logical and mathematical entities. Perhaps the classic example of an artificial paradigm shift is mapping data from a hierarchical file structure to a relational database.

A synthetic paradigm is where you take a common natural or artificial paradigm and modify it. A good example is a spreadsheet. Naturally, it represents a sheet of paper with rows and columns where you can write numbers. But then we add the ability to put live formulas in the cells. Something we can’t do in nature but a snap in the abstract computer world, and it changes everything about what spreadsheets can do.

I appreciate your interest in expressing your thoughts. This is exactly the kind of exchange I was hoping to have when I started this blog.

There are a few points you make in your response that I have a reaction to, but I’d like to focus on the one point you make about Agile practices enforcing the approach of breaking bigger problems into smaller problems.

I studied CS in 1981 when I entered college as a Freshmen. Decomposing a problem into managable components were practices we studied in college way before the Agile manifesto was birthed. On my first job in 1985, I programmed in assembly language, PL/C and Pascal. Objected oriented programming was still an idea in academia back then. During those years we did big up-front designs, but we functionally decomposed them and assigned smaller functional components out to team members to develop them in more detail.

Good teams have been decomposing problems into smaller managable components for many years. So I wonder why do the Agile proponents insist they’ve developed something new when us old timers have being doing this stuff for decades, and why do they feel they’ve created new laws of software development when these practices are anything but new? These are not concepts brought to the table by Agile; Sorry, there’s nothing inherent in doing big up-front design and analysis that prohibits or deters functional decomposition. In fact, it’s embraced by traditional practitioners, and it’s been embraced for decades. Functional decomposition is simply a good problem solving technique no matter the discipline or production methodology.

Really this whole Agile movement feels like some twilight zone episode where some kid in a remote village sees a boulder rolling down a mountain and gets the idea for a wheel, and he thinks he discovered something new.

I also agree than every problem is potentially unique. But yes, it seems we have different default approaches to software design. As you mentioned, perhaps it’s because of different experiences. So I am *not* saying one is better than another.

But it might be interesting to try and sharpen these distinctions.

Let me reiterate that I don’t consider most business software to be very difficult — rather it tends to be complex. Business software is some of the most complex software there is. Say millions of lines of custom Java code on top of some proprietary framework on top of the J2EE. Business software is much more complex, but far less difficult, than say NASA’s trajectory software for getting our latest Mercury probe (MESSENGER) into orbit around that tiny planet (by using a series of gravity assists from Earth, Venus, and Mercury itself). Easy to visualize the solution, hard to actually do.

Now, our workhorse method for managing complex problems is divide-and-conqueror. We turn a big complex problem into a bunch of smaller, simpler problems. Agile methods naturally tend to enforce this approach. Big up-front analyses do not naturally enforce this approach. (Although if the big up-front analysis is also object-oriented…that’s a different discussion.)

So I tend to favor an Agile approach by default. And thus, I tend to favor working code over design documentation. I would usually rather build upon a bunch of working code pieces than implement all new code from a design specification.

Second, I trust working code more than verified design documents. Software quality is very difficult to obtain. There is significant risk in even verified and validated designs. As they say in science: Theory is fine, but the sole test of knowledge is experiment. There is no substitute for working code. And again, this is something Agile methods tend to naturally enforce, but that big up-front analyses do not.

Thanks for reading and sharing your point of view.

I don’t agree a proper up front investment implies a complex solution, and I don’t agree that risk is reduced because of a shorter interval.

I do make the point in the essay that for some problems it doesn’t make sense to have a heavy up front investment. I advocate the up front investment to be as long or as short as required to do the proper job within other necessary constraints.

A short investment in up front analysis doesn’t guarantee a simple design anymore than a large investment in up front design guarantees a simple design. Simple designs happen when we have a good understanding of the problem, and we have the right people working on them. Smart, experienced people develop simple solutions and they arrive at them quicker than most. Having a good understanding of the problem usually requires an investment in time to understand it well, and the time required to understand the problem well increases as the problem to solve is bigger and/or more difficult.

My reference to large up front investment is to contrast it with the inadequately small up front investment advocated by Agile proponents. I don’t believe we can do a proper job by time boxing design or artificially setting a week or month iteration to deliver a feature set. I’m all for a week iteration if it’s adequate time to solve the problem properly.

Our experiences seem to be very different. The overly complex and bug ridden software that I’ve seen has been the result of a solution developed quickly to get something out the door quickly and then fixed quickly to keep it going. As future releases are built on top of the original architecture, the software further decays and becomes more complex as solutions are further coded into a fragile architecture because there is no time in the schedule to rework the original solution, so the developers have no choice but to solve new problems in more complex ways as new requirements are levied on the system. Usually in these environments there is no time to rework what could have been avoided with some more up front design and analysis — enough to be successful, no more, no less.

Great people investing the proper amount of time in design and analysis produce the best solutions.

I do accept your position that we should be striving for simple solutions, but it often takes time to arrive at a simple solution, often more time than to arrive at a complex solution.