Simple by Design
Achieving Agile goals of delivering faster and responding to change quickly is best achieved by designing an architecture that leverages those goals, I reasoned in an essay titled “Agile Isn’t a Process“. Being agile is about leveraging the entire organization and business partners to deliver solutions to your customers. Agility is best achieved when solutions require less IT involvement, not more.
While having a good development process is important and necessary, the Agile process is not well suited to developing agile architectures; an emergent, iterative approach does not offer much when the requirements for solving the problem require an investment in up-front analysis and design. Some would call the investment big, but I would call it proper. In this essay, I’d like to offer a real world example to support that original thesis.
Criticisms
It’s interesting to read some of the criticisms of the article. One reader suggested that an investment in up-front analysis and design results in overly complicated solutions. George Crews commented, “It has been my experience that taking a long time to architect software only guarantees complex software, not software that will actually work.”
It’s a common opinion. I actually agree with this comment if taking a long time is evidence of having an untalented team working on the problem. I have found the weaker performers always seem to take longer and develop more complicated solutions. On the other hand, if the team is talented, taking a long time is evidence of a difficult problem, and difficult problems require an investment in time for even talented people to solve well and simply.
Another common criticism is that short iterations reduce risk. Well, it depends. If the iteration is designed to deliver less, then sure, risk is reduced. On the other hand, if the short iteration forces the developers to cut corners to deliver within the iteration, risk is increased as requirements are missed and error conditions aren’t considered in the interest of delivering something quickly.
It’s also suggested in some blog articles that investing time in a simple but thoughtful architecture compromises present requirements. Big Design Up Front (BDUF) steals time from delivering on the more pressing present requirements. It’s said that present priorities require precedence over future needs, and therefore, it’s suggested to plan just enough for the future. However, if you expect the product that you are building to have longevity, then the future is a present requirement that should not be ignored.
Of course, the future has to be balanced with the present; there is no formula for this. Arriving at the right balance between present and future requirements is an art; it is situational (sometimes future requirements have higher priority and sometimes they don’t), and striking the right balance is usually a sign of a vibrant team, a team where contrasting ideas and opinions are plentiful
There’s another criticism in the present versus future argument. The position is that analyzing and implementing solutions that anticipate the future requires more time. My experience doesn’t support this position. Yes, analyzing and designing a good solution often takes more time to deliver that first working build, but the time lost in getting to that first build is often compensated in delivering the overall solution faster.
Another suggested reason for delivering a working solution faster is that it reduces risk since working code proves out the design. Risk is increased when the production of working code takes longer, since a flawed solution will be found late in the release cycle, where rework will impact the schedule. This position is specious. Up-front analysis and design is a risk mitigation strategy; it doesn’t increase risk. I’ve never seen a design where I could not determine whether it would be successful or not.
BDUF Example
In 1991, I was assigned to develop the solution to realize the requirement to internationalize our product. This product was known as Reuters Terminal. It was essentially a financial browser for navigating and displaying Reuters’ real-time financial news and information. Reuters financial network disseminates financial news and information from every market in the world to every financial data center in the world. Consequently having a product that was in the native language of the local market would be a competitive advantage - especially at this time when internationalization support in the Windows operating system was just being conceived.
The culture of this team was to do what is pejoratively called today Big Design Up Front, but to us, it seemed like the sensible way to design a good solution and to deliver it to market quickly. Project initiation to release was approximately six to nine months, and there were many requirements in addition to the internationalization requirements that were delivered in this product release.
Design
The architecture was a 3-tier architecture: The top tier is the application requiring the internationalization support. The middle tier abstracts the details of each language instance from the application, and the bottom tier is a language driver that captures all the details and nuances of rendering an instance of any particular language for the application. The application resources are linked into the language drivers. Essentially all the Windows APIs for rendering text, dialogs, and menus are replaced in the middle tier. Below is a pictorial representation of the architecture.
Benefits
While the first build may have taken longer than a build if this was implemented with an Agile approach, this approach has considerable advantages over the impetuous need to have a quick first build. The advantages were.
- Increases implemenation parallelism. After the interfaces for the language API layer and language driver layer are designed, the implementation can be accomplished in parallel with one or more developers working in the application layer, one developer working in the language layer, and one or more developers working in the language driver layer. The implementation is highly scalable.
- Anticipates future requirements. The application is to be customized for many languages, and this approach permits support for many languages without obfuscating the code with many macros to selectively link the application for any particular language. It permits new languages to be released without releasing new versions of the application. And it permits internationalization to be outsourced without much involvement from the core application team.
- Provides the capability to switch between languages at run time. With this architecture the application can switch between languages at run time. This is in part possible because of the architecture. The application works with the middle tier only, and switching between languages is simply having the middle tier dynamically link to one of the language drivers. Plus, there’s essentially very little extra code to support that feature. It requires the middle tier to activate a different language driver, and the application forces each of the windows on display to repaint.
- Makes software configuration management practices easier. Since at the application layer the code behaves the same no matter the language, there is no need to merge code while additional languages are in the process of being supported, and since support for new languages are delivered independently of the application, there is no need to manage and control many different versions of the application.
Once the internationalization architecture was completed in that first release, the architecture was in place for the lifetime of the application. Re-factoring is rare when problems are analyzed well.
Agile Example
An Agile team has one primary goal: deliver working code in a short iteration. If these are my goals, I would immediately begin tackling one aspect of the user interface that I can readily begin to internationalize. Perhaps, one would start reworking all the text writing code in the application to display multilingual. To begin seeing results quickly, the easiest approach would be to statically link a different resource file. Then, rewrite the application code wherever LoadString and TextOut are used. To deliver the solution quicker, one or more developers would be assigned different application modules to internationalize. While this approach would deliver working code fast, it has some disadvantages.
- Code is obfuscated with macros. To update the application code for each language, the logic for each language would be selectively compiled with macros to enable the compilation of the code for each specific language. When enough languages are supported, the language code overwhelms the primary logic in those procedures.
- Each module would likely solve the problem a little differently. By implementing the solution inline, there is more variability in the implementation when multiple programmers are working on each module. Even if the same developer were to fully implement the entire solution, there is likely to be variability in the algorithm as the developer converts each module. This makes the product more difficult to maintain.
- Quality is lower. Since the internationalization is completed inline rather than via a one-for-one Windows API replacement, a defect in the inline code may need to be fixed in every place that it’s been re-engineered, but a defect in the API only needs to be analyzed and fixed once. Additionally, there’s more of an opportunity to introduce a different defect in every place that the inline code is touched.
- The implementation is less scalable. It’s difficult, if not impossible, to outsource the internationalization development as code merging would be a requirement. There is even less ability to parallelize development when the same file needs to be re-engineered for each language to support.
- Adding new languages is harder. As new languages are supported, there will be constant merging of files with work going on to support other features.
- Configuration management is more complicated. Each supported language requires a separate build of the application, and therefore, a separate executable is required for release. Also, a bug in one language may not be present in another language, so the code base will be changing out of synch with the executables deployed.
This is certainly a viable solution with distinct and troubling disadvantages for moving the code base to the future and responding quickly to changing requirements. Unnaturally forcing a short iteration has disadvantages. We used to have or own pejorative name for this: BFM, Brute Force Method. Shortchanging up front analysis and design forces a BFM solution.
Summary
When I was implementing this solution with up-front analysis and design, we didn’t use the words Big Design Up Front. We called it good analysis and design. Sure there were engineers who spent too long in analysis and design and arrived at overly complicated solutions, but this had nothing to do with the approach: it had everything to do with the personality of the individual solving the problem. Placing the blame on BDUF is a case of mistaken identity.
Proper up front analysis and design is required, and yes, sometimes it takes longer than we’d all like it to take. However, when it’s done well, architectures are less complex, not more; quality is higher; time to market is faster, and products and teams are more agile. Proper architectures leverage the resources of the entire organization to deliver solutions to customers.
(No Ratings Yet)
Loading ...
My impression is that a lot of people claim to be “doing Agile” as an excuse for allowing themselves to do things in a “cowboy” fashion. TDD is a good tool, but no substitute for a well-thought out architecture. (I agree with you that up-front Architecture and high level Design is a good thing - http://blog.perfectapi.com/2008/01/agile-needs-architecture.html).
However, I think you have set up Agile as a straw-man in your argument. Specifically your assertion…
“An Agile team has one primary goal: deliver working code in a short iteration”
…is not true. This is taking a common Agile *practice* and presenting it as representative of Agile *values*. (It is not even a principle of Agile, although it is similar to the principle of “Deliver working software frequently”).
It is a common misunderstanding that Agile suggests up-front architecture is unnecessary. It is always a good idea to plan your architecture - Agile just advises that you do *no more* than you need. This is a direct consequence of valuing working software over comprehensive documentation and responding to change over following a plan.
Hi Bill,
Who would try to apply an Agile approach as naively as described? I have some I18N experience and can imagine taking an Agile approach to successfully implementing a solution. Perhaps my imagination is faulty, but I find this particular example unconvincing.
But I agree with you that, as put by Fred Brooks in “No Silver Bullet”: “The hardest single part of building a software system is deciding precisely what to build.”
Rather, the point I would like to make is that something being “hard” and something “taking a long time” is not, in this case, the same thing. This is the key to our difference of opinion. In my opinion, a good software design is less like running a marathon and more like high jumping. Either you clear the bar or you don’t. Either way, success or failure doesn’t take that much time, regardless of the height that needs to be cleared.
And you need the people to do it. Three high jumpers that can clear 2 feet can’t take the place of one high jumper that can clear 6 feet if that’s the height needing to be cleared.
(I will concede that developing the skill, knowledge, and inspiration necessary to conceive of a good design can take some time. So can the documentation, validation and implementation of the design.)
Hi Steve,
I’m not sure why a methodology would pick the name Agile if it weren’t chosen primarily to represent the definition of the word. As dictionary.com defines it, Agile is defined as “quick and well-coordinated in movement.” The principles behind the agile manifesto has as one of the principles to “Deliver working software frequently, from a couple of weeks to a couple of months, with a preference for the shorter timescale. So I have to believe being quick is a very important goal of Agile.
Some solutions don’t lend themselves to deliver working code in a couple of weeks to a couple of months. We didn’t have a working build for at least 2 months. Now how does a methodology such as Agile permit a team to deliver a solution that requires that kind of up front investment? Don’t tell me that all problems can deliver working code in an iteration of 2 months or less because that’s just nonsense.
Now here’s the problem with Agile: you can read many blogs with supporters of the Agile processes, and they will all have a somewhat contradictory definition of what Agile is. It reminds me of the line in “The Incredibles, if everyone is special, then no one is special.” Likewise, if every practice is Agile, then nothing is Agile. The Agile definition is too vague. There are no objective criteria for deciding whether a practice is agile or not. So we live with this constant controversy and confusion. If the Agile community wants to gain some respect, they need to be clear about what Agile is and isn’t with objective criteria. It will help the Agile community and help it gain support with those who haven’t bought in yet.
I also agree with your statement, “My impression is that a lot of people claim to be “doing Agile” as an excuse for allowing themselves to do things in a “cowboy” fashion.” This is a consequence of a vague definition. You don’t have this problem with CMMI. It has very objective criteria for determining whether a team meets the standard. Agile needs a similar definition.
Hi George,
That solution may be naive to you, but I bet many teams have solved that problem with that approach, and my experience is that human nature being what it is, most teams will go straight for BFM when they feel the pressure to deliver — I’ve seen it repeatedly. To the extent that Agile adds to that pressure with short iterations, you will see teams solve problems in a BFM way. Not all of course, but the process encourages the behavior. That’s all.
But I’m really starting to get confused as to what Agile is because the Agile proponents don’t have a consistent message, and maybe that’s the problem, the Agile proponents aren’t really sure either of what it means to be Agile.
But the real message in the essay is that agile is best achieved through proper designs and architecture. Even if you believe the contrasting solution is ludicrous, there is nothing a development process can do to rescue it.
Now if any Agile proponent observed our practices developing that problem, they would call us troglodytes. This was BDUF, it was waterfall, not in the traditional sense that Agile proponents like to caricature it. Yet we were very agile with a small “a” because our solutions permitted a high degree of scalability in implementation and because you can deliver smart solutions faster than naive solutions. We always aimed to deliver smart.
Now we came up with a simple, elegant architecture with an approach that you said would only lead to complicated solutions. How do you explain that?
Hi Bill,
I hope you weren’t offended. That certainly was not my intent. I would like to point out that I did not say that the described Agile design was ludicrous — rather I used the word naive. I agree with you that many teams have used even worse approaches to similar problems.
Do Agilistas really know what they are talking about? I would think so. The Agile method is DEFINED by the Agile Manifesto (http://en.wikipedia.org/wiki/Agile_Manifesto) which is a pretty succinct document . But I am not surprised you don’t think I understand it too well. I often don’t exactly know what I am talking about. I have to change my mind a lot to keep on the right track.
The Nokia Test (http://djellison.blogspot.com/2007/12/nokia-test-are-you-really-agile.html) has recently been put forward by several people as objective criteria for determining if an Agile method is actually being performed. It too is pretty short.
I agree with you that Cowboy Coding (http://en.wikipedia.org/wiki/Cowboy_coding) is less likely using BDUF methodologies since they discourage immediate coding efforts.
I was hoping you would allow me an exception or two on my observation about long-time-interval designs being complicated. But I think I can go more than one-to-one on examples vs. counter-examples: the FBI’s Virtual Case File project being an “outstanding” one. (http://en.wikipedia.org/wiki/Virtual_Case_File)
Finally, I would point out that CMMI is not a process guide, but a process improvement guide. It contains no software engineering methodologies. Also, its empirical foundations are just as flimsy as Agile’s.
Hi George,
I’m not at all offended. I value good honest exchange of views – even if they are rough. I do realize you didn’t use the word ludicrous. I was taking your naïve view to an extreme to make a further point.
My problem with Agile is many folds, but mostly that the focus is on the wrong things, and I believe at a minimum this essay makes the point: agility with a small “a” is best achieved via good solutions, solutions that are extensible and make it easy to change the working application. No process methodology can rescue that contrasting solution in the essay: not a waterfall based methodology nor an Agile based methodology. Small iterations don’t rescue a bad solution. The quantity of documentation (little or a lot) won’t rescue a bad solution. Agility is foremost achieved through good designs, implementations that rarely require re-factoring.
The Agile Manifesto doesn’t make a single point about the need to create good solid designs and architectures. In the Principles Behind the Agile Manifest, the word “simplicity” is used not in reference to architecture and designs but to do less work. Only one small point in the Principles Behind the Manifesto about good design: “Continuous attention to technical excellence and good design enhances agility.” Now the Agile guys can caricature the BDUF approach favored by the so called waterfall proponents, but make no mistake the emphasis of the traditional practitioners is on design and architecture and the Agile camp is ridiculing them for emphasizing the right thing. Agile has the wrong emphasis.
I’m never going to suggest that complicated designs won’t be developed by traditional practitioners. Of course, they will. Complicated designs are a function of the caliber of people. It’s not a methodology issue unless you have a methodology like Agile that encourages going to implementation too quickly. I’ve read blogs of Agile proponents who promote that analysis, design and construction happens in parallel in Agile. How? How do you write code for a problem where you’re doing analysis and design concurrently? It’s silly.
You are mistaken about CMMI. It’s a process guide. It identifies KPAs (key process areas) and the objectives for satisfying compliance to the goals of the KPAs. It’s up to the implementers to define the process. It’s a much more rigorous and objective form of the Agile Manifesto with a lot of Greenspan like obfuscation. The obfuscation is both its blessing and its curse. It’s a maturation model in that you build your process like building a multi-story building: one floor at a time. The challenge is to develop a lean process while meeting the objectives. Too many teams that use the model go overboard. They miss the essence.
I’ve been involved in 3 CMM(I) process initiatives and heavily involved in developing one of them. I would say only one of the 3 processes that I was involved in would I consider good. Not sure what you refer to with the empirical foundations, but there is no doubt that a company has level 2 practices. There are very objective criteria for establishing that conclusion. This is missing in Agile, and it’s why even the Agile proponents seem to be at odds as to what Agile is.
Hi Bill,
You raise several good points that all deserve discussion. Let me respond here only to the main point.
I agree that no software methodology (Agile or BDUF) can rescue a bad solution. And I also agree with you that, unlike BDUF, the Agile methods do not emphasize a process for designing a good solution. But perhaps, as we say, that’s not a bug but a feature.
The advantage of using Agile methods is that they identify bad solutions more quickly and more definitively than BDUF methods. They also embody a particular promising approach. This allows better risk and project management.
Maybe it would help to know where I’m coming from. I have a science and engineering background whereas most of my fellow software engineers I believe have a math background. I’ve noticed many mathematicians live in a Platonic world. I note many scientists and engineers do not. Most of my cohorts tend to think that if a software engineering process is logical and consistent in handling a problem’s requirements, the process will work. But I don’t. (I also don’t think a sense of conviction has anything to do with reality. Just because my cohorts think they have a great design doesn’t mean they actually do.)
The attraction of Agile methods is that at the end of a sprint, a relatively short time, you will know (not think, but know) if a particular approach won’t work or needs major modification. Logic and analysis can’t do this, no matter how formalized a process is used, for the same reason scientific theory has to rely on experiment.
Hi George,
I agree that there is something to be gained by having a working application and getting there as soon as possible, but traditional practitioners have been doing that since the 80’s when I first started in this field. It’s not new. What is new is to artificially timebox that first iteration. Whenever I plan a project, I look to identify when I can have that first build, but I let the work dictate the iteration, not an artificial time frame.
Where I’m confused is this idea that you cannot know if a design will work or not until it is implemented. I was fortunate enough to be mentored by really smart guys when I first started out who could rip apart a design, any design. They knew if it would work or not before it was coded. This saved time. If I had coded the work that I designed without the redesign that they recommended from reviews, all the design mistakes would have been caught at unit testing where it would have taken much longer to refactor and get working. I’ve never reviewed a design where I didn’t know whether it would work or not at review time. Reviews are valuable, and good reviews are necessary to be agile.
But really, what does seeing something work early tell us that’s valuable about the design? If an Agile team had implemented my contrasting example, they would have seen working results in the first sprint, but it still would have been a lousy design and implementation. Working is NOT evidence of a design being qualitatively good. The Tacoma Narrows Bridge worked, but it was a lousy design, and it finally collapsed. Sometimes knowing something is a lousy design takes time when you rely on only observational based evidence of it working to conclude it is good.
In the case of the contrasting example, a good review by good engineers would have provided more benefit than a working first sprint. Time well spent in design and analysis would have provided more benefit than a working first sprint. I’ve managed large software projects, and whenever I’ve given into the pressure to deliver something sooner and sacrafice sufficient design work, it’s always been a mistake.
Leave a comment!
EMAIL
Subscribe
Visitors
Polls
Categories
Blogroll
Tag Cloud
Agile Analysis BDUF Certifications CMMI Critique Debate Defect Free Design Estimating Ethics excellence Future good enough Hiring Leadership LOC Management Methodology Metrics Outsourcing Popular Process Product Management Project Project Management Project Tracking Quality Recruiting Refactoring Requirements Schedule Scrum Skills Stable Test Strategy Tracking Traditional Upfront Video Waterfall wrap-up XPRandom Posts
Latest Video Post
Recent Comments
Most Commented
Most Viewed