The term predictive is deceptive.
In the Disciplined Agile (DA) tool kit we use the term traditional or serial, rather than predictive or waterfall, to refer to the classic/linear ways of working. We feel that predictive is deceptive, more on this in a minute, and waterfall to be insulting (albeit still in common use within the IT community). Furthermore, we're starting to move away from using traditional as we're now seeing a generation of practitioners who feel that some of the older agile approaches, in particular Scrum, are traditional ways of working.
There are several reasons for why we feel the term "predictive" to be deceptive:
In short, we know that "predictive" is a deceptive term for a large category of projects and suspect this to be true for other project types. As a result the only use of the term predictive in DA is to tell you that we don't use it.
The Disciplined Agile (DA) tool kit is organized into four layers: Foundation, Disciplined DevOps, Value Streams, Disciplined Agile Enterprise (DAE). This blog focuses on the Foundation layer, the purpose of which is to provide the underpinnings of the DA tool kit. The foundation layer itself is organized into four distinct topics:
The DA Mindset
The Disciplined Agile (DA) mindset is captured in the form of principles, promises, and guidelines. Disciplined agilists believe in the DA principles, so we promise to adopt these behaviours and follow these guidelines when doing so. There is a purpose for each aspect of the mindset:
Disciplined Agile (DA) is a hybrid in that it adopts ideas and strategies from a wide range of sources. DA encompasses three categories of fundamental concepts:
The people portion of the Foundation layer addresses two key aspects of agility:
Choosing Your WoW
A fundamental philosophy of agile is that teams should own their own process, or as we like to say in Disciplined Agile (DA) teams should choose their way of working (WoW). Of course this is easier said than done in practice. The challenge is that every team is unique and faces a unique situation – in other words, context counts. Furthermore, there are no “best practices,” rather, every practice has tradeoffs and works well in some situations and poorly in others. Worse yet, you really don’t know how well a technique will work for you until you actually try it out in your environment. Given all of this, how can a team choose its WoW?
While working with organizations to help them to learn how to improve their WoW, we’ve developed a technique that we call guided continuous improvement (GCI). GCI extends the kaizen-based continuous improvement approach, where teams improve their WoW via small incremental changes, to use proven guidance to help teams identify techniques that are more likely to work in their context. This increases the percentage of successful experiments and thereby increases your overall rate of process improvement.
We’re often asked by people when it makes sense for a team to take a traditional approach to software development, and more generally when it makes sense on any type of project. While the focus of this article is on software development projects it also discusses non-software projects too.
Sometimes people are honestly trying to identify when each potential lifecycle, including the traditional lifecycle, makes sense to follow. In many cases this request is coming from people who are desperate to continue working in the same old way that they’re comfortable with. They often hope that the context of the situation that they’re in puts them in a position where they don’t need to adopt new ways of working. Some people get “lucky”, although how lucky it is to forgo an opportunity to gain new skills that are currently in demand is questionable at best, but most find that they need to join their colleagues in adopting agile.
Traditional development was prevalent in the 70s through 90s, but starting in the mid-90s organizations started to adopt iterative approaches such as the Unified Process (UP) as well as agile approaches such as Scrum or Extreme Programming (XP). Having said all this, the majority of organizations still have a few teams following traditional lifecycles and will often have people who believe that traditional software development is a good idea. And, as I argue in this blog, in a few situations they’re right about that.
This blog explores several issues surrounding the traditional approach:
1. What is Traditional Development?
Traditional development - also called serial, waterfall, linear, or "predictive" - is based on the idea that the delivery lifecycle should be organized into phases or stages. Figure 1 depicts a classic waterfall lifecycle, similar to what Winston Royce originally proposed (and recommended against) in his 1970 paper. In a pure waterfall the flow is unidirectional (from Requirements to Architecture to…) whereas Figure 1 depicts a “modified waterfall” using bi-directional flow that indicates feedback from a phase may go back to the previous phase.
Figure 1. The Waterfall lifecycle.
Figure 2 depicts a gated waterfall where you need to pass through a “quality gate” to move between phases. The “quality gate” tends to be based on the review and acceptance of artifacts – for example a Software Requirements Specification (SRS) coming out of Requirements, a Software Architecture Design (SAD) coming out of Architecture, and so on. It also shows how feedback can go back to any phase via a Change Control process, which on most traditional teams proves to be a change prevention process in practice. I used quotes around “quality gate” because “quality gates” tend to have very little to do with quality in practice and have a lot to do with promoting questionable bureaucracy. My point is that traditional rhetoric may be blinding you to serious deficiencies in this approach.
Figure 2. The Gated Waterfall lifecycle.
Figure 3 depicts the V-model version of the traditional approach where the test phase is depicted as multiple stages and there are clear indications of what each testing activity validates (for example, Integration Test validates your architecture).
Figure 3. The V-model lifecycle.
The governance strategy with a traditional approach tends to be artifact based (i.e. someone reviews and signs off on your architecture document) rather than risk based (i.e. we proved that the architecture works by building a vertical slice of the solution that implements high-risk requirements). Each phase tends to be staffed with specialists (i.e. requirements are captured by requirements analysts, the design is created by designers, and so on) which tends to motivate a quality gate approach, the development and maintenance of traceability information, and significant management effort to coordinate everything. This in turn increases cost, overall cycle time and decreases ability to support changing requirements (traditionalists typically fear “scope creep” whereas agilists embrace evolving requirements) which in turn decreases stakeholder satisfaction with the end product.
2. When Does Traditional/Waterfall Make Sense?
There are several situations when the traditional approach makes sense:
3. What Factors Have No Bearing on When Traditional Makes Sense?
My experience is that the following issues do not have an impact, or when they do have very little impact, on the deciding if a traditional strategy is appropriate for your team:
4. But Isn’t Traditional Better at Scale?
Not really. Agile and lean are as good or better when applied by large teams, geographically distributed teams, teams in regulatory situations, teams facing domain complexity, teams facing technical complexity, and teams facing organizational distribution. Figure 4 depicts a radar chart of potential tactical scaling factors.
Figure 4. Scaling factors faced by software development teams.
So how does traditional fair in comparison with agile and lean strategies in practice? If you poke around at the IT Surveys page you’ll see that the 2014 Software Development at Scale study found that agile teams outperformed non-agile teams across a range of success factors. Similar results can also be found in the 2006, 2008, 2010, 2011, and 2013 Project Success Surveys. The Dr. Dobb’s Journal 2008 Project Success study found that when comparing agile and traditional approaches based on level of geographic distribution that agile teams were at least as successful and often more so, on average, than traditional teams. In other words, agile was less risky. In 2006 DDJ found that agile was as good or better than traditional regardless of team size, but unfortunately I can no longer find those results online.
To be clear, some of these surveys are a bit long in the tooth. Having said that, in the past few years organizations have figured out how to successfully apply agile and lean approaches at scale. I suspect that agile and lean will both have pulled ahead even further compared with traditional at scale. We’re actively looking into these sorts of issues so stayed tuned to this blog for future research results. And, we’re not the only ones who are getting these sorts of results. Go poking around on the web and find out for yourself.
5. Does Disciplined Agile (DA) Support the Traditional Lifecycle?
The DA toolkit does not yet support a traditional lifecycle, although an upcoming release in Q2 2020 will bring some aspects of serial/traditional into play.
Having said that, DA has always recognized that in the vast majority of organizations you will have a multi-modal approach where some teams are following an agile approach, some are lean, some are taking a continuous delivery approach, and some are still following traditional. The more disciplined your organization, the more skilled your people, the less likely it is to have traditional teams in practice.
6. Doesn’t DA Support a Serial Lifecycle?
Yes, but it’s risk-based, not artifact nor activity based as you see with traditional approaches. The two project lifecycles supported by DAD, the Scrum-based Agile lifecycle and the Kanban-based Lean lifecycle, have three phases: Inception, Construction, and Transition. These phases, overviewed in Figure 5, capture the ideas that a project team needs to be initiated somehow (Inception), the solution must be developed (Construction), and then released into Production (Transition).
Figure 5. The phases and milestones of DAD project teams.
When you have stable (long-lived) product teams Inception tends to disappear (with the exception of the need for occasional modeling and planning sessions to think things through) and Transition evolves from a multi-day or multi-week phase into a multi-minute or multi-hour activity. Furthermore, the lifecycle evolves over time as you improve, moving from a phased agile/lean project lifecycle into a continuous delivery lifecycle. More on this in future blogs.
7. What About Non-Software Projects?
Although much of the advice presented above applies to non-software projects there are some differences in the non-software world. For example, in the physical construction world (think building buildings, oil refineries, highways, ...) they will often work in a traditional manner. This happens for many good reasons:
Having said that, even when it makes sense to follow a more traditional approach you can still benefit from applying agile strategies on your project. For example I was recently involved in a physical construction project and I was struck by how much of the actual work was performed in an agile manner. At a high-level it looked like a traditional/serial project, and it was, but on a day-to-day basis it was reasonably agile within the limits they had. So it was more of a hybrid strategy masquerading as traditional.
8. Why Should You Listen to Me?
Here is a brief description of my background in traditional software development:
The point is that I believe I have sufficient experience with traditional approaches to speak about where it does and doesn’t work. I hope this blog has provided some valuable insights for you.
Yesterday Mark and I double teamed on a webcast overviewing DAD to a university class. After the webcast we got the following question:
I am wondering if it would make sense to entirely eliminate the word “phase” from DAD’s vocabulary. What about simply talking about different types of iteration? For instance, inception could become something like “pre-construction iterations”, construction could become “construction iterations”, and transition “transitions iterations”… That may be a bit cumbersome, but the word phase does scare people away.
Yes, the word phase might scare some people away. We’ve thought long and hard about the terminology that we use and were also a bit worried about the word “phase” at first. For some reason phase has become one of those dirty words in the agile community, along with other words such as governance or documentation. The primary concern seems to be that phase implies a serial approach, something many agilists believe to be foul and evil. Yet the reality is that projects do in fact proceed in a serial manner. There are some project initiation activities at the start. Then there are construction iterations/sprints one after the other in yes, a serial manner. Then there is an effort to deploy/release your solution into production. This is clearly “serial in the large”.
But, why the term phase? Why not iteration? Or season for that matter season (Gary Evans has a very coherent argument for that term)? Because phase is accurate (albeit an agile swear word). In the various IT surveys that I’ve run over the years I’ve found that the average agile team spends about 4.5 weeks performing initiation efforts, has construction iterations that are about 2 weeks in length, and takes an average of roughly 4 weeks to release their solution into production. So perhaps these initiation and release efforts could each be thought of as two iterations (on average, your mileage may vary) but they really aren’t iterations in their own right usually. Maybe they’re different length iterations? There are several ways of thinking about it, but notice how the application of the term iteration doesn’t really make perfect sense. Then we have numbering issues. Is the initiation effort iteration/sprint 0? If it’s two iterations would they be -1 and 0? Sigh.
One thing that we have done which some may feel to be radical is we’ve chosen to adopt phase names from Unified Process (UP) – Inception, Construction, and Transition. We could have made up different names, such as Initiation, Development, and Release respectively. Or adopted Start Up, Construction, and Hardening (yuck, there’s more to Transition than hardening and frankly I would consider a late “hardening” effort to be a clear sign you’ve likely got a process problem you need to deal with). And I have no doubt that there are many other options for each phase name. Whatever names we choose someone isn’t going to be happy, and why not give a bit of a nod to a proven software development framework such as UP?
Another interesting thought is that by having named phases it makes it clearer where potential governance points in the lifecycle are. In the diagram below you can see how several of the suggested light-weight milestones – Stakeholder vision, Sufficient functionality, and Delighted stakeholders – corresponds to the end of a phase. Or perhaps more accurately, fulfilling the milestone is an indication that your team is moving from one phase to the next.
In the end, I think we’re pretty clear that when you tailor DAD that you can use whatever terminology you like. In fact one financial institution that adopted an early version of the DAD basic lifecycle, the one based on Scrum, reworked the diagram to use Scrum terminology. I think it’s a bit goofy but it works for them.
The term phase might in fact scare a few people away. But, it’s hard to imagine that anyone that concerned about what something is named will be flexible enough to be successful at DAD anyway.