June 24, 2003
"There is a theory which states that if ever anybody discovers exactly what the Universe is for and why it is here, it will instantly disappear and be replaced by something even more bizarre and inexplicable. There is another theory which states that this has already happened." -- Douglas Adams
Bathing under a waterfall?
"The granddaddy of all lifecycle models is the waterfall model," noted Steve McConnell of Rapid Development in 1996. The waterfall approach is most certainly the granddaddy, and maybe that is why so many project managers and team leaders tend to stick with it even when it's inappropriate; it is the devil they know. Well hopefully it's this risk-aversion of project managers and not their inexperience that leads them, like Linus, to hold their comfy blanket tight.
We're not saying it's appropriate for all projects, but an iterative and incremental development approach will reduce risks, improve schedule time and lead to a better end product. Yeah, those are not our words--we have all read them in many project management books. What we will do in this article is put the spotlight back on the iterative approach to help managers identify when it is applicable, explain why it works and at the same time discuss the role that key members of the team can play in an iterative methodology.
What's wrong with granddad?
There are many problems with the waterfall approach. We will discuss the ones that we think are the key problems. These problems may not be limited to the waterfall approach, but granddad is best used to show up some typical problems in traditional lifecycle approaches.
First off, waterfall is heavily reliant on the requirements document. That "theoretical" set of functional requirements so lovingly prepared by the analysis team and so neatly presented in a formatted document, however, are seldom sufficient to build a system. The English language is filled with ambiguity and examples produced in Excel worksheets can never convey the full complexity of the problem at hand.
A document that is short and sweet is not likely to contain sufficient detail, while a well-written requirements document may be so long and boring that neither the users nor the development team will ever pay it much attention.
Secondly, development is almost complete by the time the users get to interact with the system. Even if the system has been developed to the exact standards of its requirements document, we all know that users change their requirements daily and will have a million and one new requests as soon as they see their cool new system in action. More likely--and a more significant risk--is that the system does not meet users requirements as they envisaged and a substantial part of the code needs to be thrown away and rebuilt.
Finally, with large new systems, developers are required to perform research and development, not just development. Developers have to investigate uncharted territory and come up with performance, functionality and interaction solutions that live up to new expectations. Under the waterfall approach there is no space for research. The whole system gets built in one large bulk development, and if pieces don't work they will get detected too late, incurring a material cost as a consequence.
It's these problems that highlight the kind of project that would be well suited to an iterative and incremental approach. It's the kind of project where development is being undertaken for a large and complex system, where users are likely to change their requirements often and where the system being built is substantially different from anything ever built before. It's the kind of project where analysts, developers and users need a chance to experiment, test and try ideas before deciding on which works best.
That's not what I wanted!
Following some problems they were having with requirements, a friend of mine recently told me that they "had put an iterative approach in place...had broken the project down into smaller pieces instead of developing in one large chunk." A modified waterfall approach may very well solve some problems, but it is not an iterative approach.
Maybe I should have asked him how this reduced the risk of poor requirements documentation? Ambiguous statements, incomplete examples and a lack of investment in requirements gathering can all lead to that all-too-common problem of users not getting what they expected.
An iterative approach allows us to start building the system without having a detailed description of the end-state. Armed only with a set of high-level requirements, analysts can begin to build system prototypes. "Did I hear you right, did you say analysts"? That ground has already been covered, so we'll refer you to our previous gantthead article titled "Degrees of the BA".
An iterative approach forces a bigger investment in analysis. It couples the users, requirements and design into one single and cost-effective process. A prototype will demonstrate exactly what the system looks like and does. This allows the users to provide immediate feedback and think about their future requirements. If things go wrong, then the loss of code does not result in a material cost. If things go right, then the prototype can provide a much better building block than a Word document. In addition, each and every person involved in prototyping moves up the learning curve and is better placed to deliver a better quality system.
The final prototype may not contain the world's best code or architecture, but it gives all project participants a more complete view on the system requirements and a full understanding of technical implementation issues. It's crucial at this stage that the prototype gets reviewed and improved. Too often the prototype is put into production as is. Before it goes into development, the analyst and architects--armed with a better picture of what the system will look like--can come up with a design of good quality. At this stage, project participants cannot be fearful of throwing away parts of their past work.
Stop moving the goalposts!
"Those damned users! They keep on asking for something else." When users see a new system, new ideas will be sparked and requirements will grow. From research comes new ideas. Chemist Roy J. Plunkett discovered Teflon while researching refrigerants. Similarly, we may discover new uses and ideas for our system as we prototype and present the system to our users.
When a business stops caring for its customers, it will most likely go out of business. So are we any different? Instead of complaining, we must service our clients needs. Requirements will change. It leads to a better product, and we should not try to prevent it. What we should instead do is adapt our methodologies to encourage new ideas and system improvements. Obviously we need to keep an eye on cost vs. benefit and scope creep, but we should do this in a way that does not impede innovation.
An iterative approach is well suited to moving goal posts. We only kick the ball a little distance, and when the goal posts move we get to run after the ball and kick it again. The difficult part in this approach is choosing small but functionally rich pieces of code to prototype. We want to elicit maximum feedback from the users and at the same time provide the design team with enough detail to start building the foundations without tackling too much of the build.
The analyst's role is key. It is up to him/her to get the users participating in the process and to get them excited about their new system. As we mentioned in our previous article, the analyst should have the ability to build this prototype. This will cut down the costs of prototyping since a smaller set of people can perform the analysis and prototype build. Once the prototype reaches the point where a system is beginning to form, the developers and architects will get involved to improve design and code quality. This whole process brings users, analysts and developers much closer together, reducing misunderstanding.
It should look like a triangle, behave like a square and move like a circle
We are all too keen to think that we have seen it all before and that what we are building is just a tweak on another system. We may be right half the time, but that means that half the time we are wrong. What we might be building on this occasion is new and has never been seen before. Its time for R&D.
To iterate is to research, to increment is to develop! It may seem like a bit of an odd phrase, but how do we conduct technology research and development? We theorize, we put a theoretical idea into practice and then we test the results. If it does not work, we throw it away and start again. This is the process of iteration. If it does work, then we take the lessons that we have learned and add to them. This is incremental.
Why is it that software developers approaching new systems and functionality do not think of themselves as researchers? We are theorizing, building and testing. We should do that at a minimal cost by using small pieces of code to iterate with. By applying a less rigid lifecycle methodology we will reduce costs because we test smaller pieces of code. When it's wrong, there is less that gets thrown away. A traditional approach applied incorrectly when we should be doing R&D will result in very large and costly failures.
There we go, it's all better now
It will take practice to perfect, but we have done it: We have successfully applied our first iterative and incremental approach. So what did we get out of it? For one, the users are happy because they got what they wanted, even though it's not what they wanted when the project started. Secondly, it cost us less because the users found fewer errors when it came to testing and we didn't have to rebuild a large part of the system. Finally, bad code was thrown away and reworked for production leading to a better quality system.
So it's time to start again. Who needs a square that looks like a circle and moves like a triangle?
Rod Pienaar works on investment banking systems as a business and quantitative analyst in the City of London and consults for www.yieldcurve.com. He previously worked in risk management consulting and has more than eight years experience in investment banking. Rod obtained his BCom and BAcc at the University of Witwatersrand in South Africa. He then moved to London where he worked as a management consultant specialising in market risk before joining Deutsche Bank in 2000.
David Van Dyke joined Deutsche Bank in 2001. David works in a project role providing systems and business advice specialising in structured equity finance analysis and process reengineering. He was previously an IT consultant working with ERP and financial systems implementations. David has obtained a B Bus (Info Systems) from Central Queensland University in Australia.