Categories: Agile
Let me preface this by saying that this is not the introduction to 10+ posts on the Flavors of the Waterfall methodology. It's one post with the intent to explore Waterfall's origins.
Why? I've lost track of the number of times I've heard (or read) Agile evangelists offer a proscriptive definition of Waterfall and then blame the process for failed projects. They almost make you think that every Waterfall project fails. So, what happened before Agile, and what, exactly, is Waterfall?
The most popular theory that I can find is that Waterfall came from a paper written by Dr. Winston W. Royce, titled "Managing the Development of Large Software Systems," published in 1970. Dr. Royce never named a methodology Waterfall, and in his paper he gives the impression that he thinks that there is more than one approach to software development.
The first approach is described as the "…two essential steps common to all computer program developments, regardless of size or complexity…" analysis and development. If the program being developed is small enough, this is all that is needed. Hmmm. This doesn't seem highly proscriptive, and could be argued to be the foundation for iterative development (although a bit of a stretch). This must not be Waterfall.
The second approach, specifically for developing large computer programs for delivery to a customer, is described with the following phases:
- System Requirements
- Software Requirements
- Analysis
- Program Design
- Coding
- Testing
- Operations
Now that looks like Waterfall!
But Dr. Royce didn't stop there. He went on to explain that each of these phases has an iterative relationship with the phase immediately preceding it. What this means is that once System Requirements are complete, it is possible that something could be found during Software Requirements that requires you to go back to System Requirements, and so on through the phases. That does not sound like the Waterfall that the Agile evangelists complain about.
He didn't express any concerns with the iterative relationship between sequential steps. The concern he expressed was that this iterative relationship had the potential to jump back multiple phases, resulting in significant delays because you couldn't skip back to where you came from; you had to go through the phases in sequence to get back to where you were. This sounds more like the Waterfall that so many have come to know and dislike.
This is also the first two pages of the paper. The rest of the paper consists of diagrams and steps to eliminate the risks inherent in the process. These steps are:
- Introduce preliminary program design before analysis
- Document the design. Dr. Royce was a firm believer in documentation, and lots of it. Yes, we're back on the Waterfall path, again. However, he is not recommending documentation for the sake of documentation. He just feels that a large software development project requires a lot of documentation, for very specific purposes.
- Do it twice. Run a smaller scale pilot, in a shorter timeframe, to simulate the process and identify then resolve problems before running the full project. And, yes, this is, counterintuitively, done before the next step…
- Plan, control, and monitor testing.
- The "last" step, although it is not so much a sequential step as a set of steps throughout the project, is to involve the customer. Three formal points for identifying the customer are identified, 1) during the Preliminary Software Review, held between the Preliminary Program Design and Analysis, 2) during the Critical Software Review, held between Program Design and Coding, and 3) during the Final Software Acceptance Review, held after Testing and before moving into Production. The expressed purpose for involving the customer is to gain commitment; giving the developer free rein is a recipe for failure.
I'll be honest, I have used similar phases but, in 15 years managing projects, I have never done it exactly like this. Does that mean I wasn't using Waterfall, because I wasn’t doing it right?
No. This process potentially provides a foundation for both Agile and modern Waterfall. On the Agile side, there is iterative work and maintaining a close relationship with the customer. On the Waterfall side, there are sequential phases and lots of documentation.
This process could take a lot of time. Keep in mind that Dr. Royce never described a process for a medium sized development project. The process he described is intended for a large software project, which could take a lot of time even if you used Agile, instead. Agile does not guarantee faster delivery of a completed project, it attempts to deliver useable features faster. Agile may actually take longer than Waterfall on a large project, with the tradeoff being that what is delivered has a higher likelihood of being useful (and different from what was designed at the beginning of the project).
So, what is Waterfall, today, 46 years after Dr. Royce's paper was published? It's still sequential, mostly. It can still require a lot of documentation, but each project should be looked at to determine how the phases should interact and what the right level of documentation is. The area where Waterfall struggles the most is an area that would cause Agile to fail, as well. Customer involvement. Agile prescribes customer involvement. Waterfall apparently does, too, according to Dr. Royce. It's just not always implemented that way. When an Agile evangelist blames Waterfall for project failure, that person is removing accountability from the people who ran the process that failed. The project that failed may be one that would have been better served by Agile, but it may also be that the process needs to be adjusted and the people running the process need to learn from their mistakes.
