Disciplined Agile

Quick answer

The term predictive is deceptive.

Detailed answer

In the Disciplined Agile (DA) tool kit we use the term traditional or serial, rather than predictive or waterfall, to refer to the classic/linear ways of working.  We feel that predictive is deceptive, more on this in a minute, and waterfall to be insulting (albeit still in common use within the IT community).  Furthermore, we're starting to move away from using traditional as we're now seeing a generation of practitioners who feel that some of the older agile approaches, in particular Scrum, are traditional ways of working.

There are several reasons for why we feel the term "predictive" to be deceptive:

1. "Predictive" implies predictable.  Predictive is defined as "relating to the ability to predict" whereas predictable is "something that happens in a way or at a time that you know about before it happens."  Something that is predictable is a sure thing, yet something that is predictive is not. This is an important difference, particularly given that we know that projects aren't completely predictable - otherwise we wouldn't need risk management.
2. "Predictive" approaches to IT projects are a poor choice in most cases.  Years ago I led a study for Dr. Dobb's Journal that investigated the effectiveness of different approaches (agile, lean, iterative, ad hoc, and traditional) to IT projects.  We found that traditional strategies were less effective in practice than agile and lean approaches, and we weren't the only ones to have found this.  We also investigated what was initially predicted at the beginning of the project and what actually happened by the end of the project, and once again traditional approaches didn't do as well as agile & lean.  BUT, I must stress that the study focus was on IT projects only, not on projects in general.  
3. "Predictive" approaches to intangible projects are likely a poor choice  DDJ found, in several studies in fact, that "predictive" strategies were less predictable in practice than agile/lean approaches in IT.  I highly suspect that this is true of intangible projects in general although do not have hard data to back up that claim.  We need to investigate this.
4. "Predictive" approaches to tangible projects are likely a good choice, but I suspect we can do better.  I suspect that "predictive" approaches are more appropriate for tangible projects, such as building houses or buildings, than agile/lean approaches.  I also believe that a hybrid approach combining the best from traditional, agile, and lean strategies is likely better than traditional alone. Having said this, as with the previous point, I don't know of any research that has compared the various project management paradigms for tangible projects, so this too is something we need to investigate.

In short, we know that "predictive" is a deceptive term for a large category of projects and suspect this to be true for other project types.  As a result the only use of the term predictive in DA is to tell you that we don't use it.

The Disciplined Agile (DA) tool kit is organized into four layers: Foundation, Disciplined DevOps, Value Streams, Disciplined Agile Enterprise (DAE).  This blog focuses on the Foundation layer, the purpose of which is to provide the underpinnings of the DA tool kit.  The foundation layer itself is organized into four distinct topics:

1. The DA mindset
2. Fundamental concepts
3. People
4. Choosing your WoW

The DA Mindset

The Disciplined Agile (DA) mindset is captured in the form of principles, promises, and guidelines. Disciplined agilists believe in the DA principles, so we promise to adopt these behaviours and follow these guidelines when doing so. There is a purpose for each aspect of the mindset:

• Principles. The principles provide a philosophical foundation for business agility. They are based on both lean and flow concepts.
• Promises. The promises are agreements that we make with our fellow teammates, our stakeholders, and other people within our organization whom we interact with. The promises define a collection of disciplined behaviours that enable us to collaborate effectively and professionally.
• Guidelines. These guidelines help us to be more effective in our way of working (WoW) and in improving our WoW over time.

Fundamental Concepts

Disciplined Agile (DA) is a hybrid in that it adopts ideas and strategies from a wide range of sources. DA encompasses three categories of fundamental concepts:

1. Agile. Agile is both a mindset and a skillset. As a mindset Agile is the manner in how you look at your environment; it is the desire to collaborate with and to learn from others; and it is the willingness to share your skills and knowledge with others.  As a skillset Agile varies based on the domain in which you operate.  For example, an agile skillset for a marketing professional may include experimental strategies such as minimum viable products (MVPs) and marketplace sensing strategies, whereas an agile skillset for a software professional may include test-driven development (TDD) and chaos engineering techniques. Agility is the ready ability to move with quick and easy grace to respond to changes in your operating environment.  
2. Lean.  Lean produces value for customers quickly through a focus on reducing delays and eliminating waste which results in increased quality and lower cost. Lean philosophies and strategies infuse DA, and in fact much of the DA mindset reflects lean thinking. This includes ideas such as optimizing flow, making all work and workflow visible, keeping workloads within capacity, and attending to relationships throughout the value stream to name a few. 
3. Serial. DA adopts great ideas from serial - often referred to as traditional, waterfall, or even predictive - ways of working (WoW). There are many strategies and concepts from the past which are critical to our success in the future, and DA chooses to leverage them where they make sense. For example, DA's project-oriented lifecycles have explicit, named phases which are clearly serial in nature. DA recognizes that inception efforts, sometimes called  project initiation or simply initiation activities, such as initial planning, initial scoping, and initial design can be critical to your success.  These efforts are very different than the construction/realization efforts that happen after this, and different yet again than the transition/delivery efforts that then follow, and different again than the operational efforts that bring actual realized value to your customers.

People

The people portion of the Foundation layer addresses two key aspects of agility:

1. Roles (and responsibilities). The DA tool kit captures a wide range of roles that people may fill.  This includes common agile roles such as Team Lead/Senior Scrum Master, Product Owner, Architecture Owner, and others.  It also includes function-specific roles such as Program Manager, Financial Specialist, Governor, Security Engineer, Sales Manager, and many others.  All of these roles have associated responsibilities, and of course there are common rights and responsibilities that everyone has.
2. Teams. Every team is unique and faces a unique situation.  As a result, DA supports several team structures which your teams can adopt and evolve to meet their needs.  There are suggested structures for small, medium, and large (program) teams.  There are team structures that support geographic distribution.  There are team structures that support learning teams such as communities of practice (COPs)/guilds and centres of excellence (CoEs), and structures that support common services teams.

Choosing Your WoW

A fundamental philosophy of agile is that teams should own their own process, or as we like to say in Disciplined Agile (DA) teams should choose their way of working (WoW). Of course this is easier said than done in practice. The challenge is that every team is unique and faces a unique situation – in other words, context counts. Furthermore, there are no “best practices,” rather, every practice has tradeoffs and works well in some situations and poorly in others. Worse yet, you really don’t know how well a technique will work for you until you actually try it out in your environment. Given all of this, how can a team choose its WoW?

While working with organizations to help them to learn how to improve their WoW, we’ve developed a technique that we call guided continuous improvement (GCI). GCI extends the kaizen-based continuous improvement approach, where teams improve their WoW via small incremental changes, to use proven guidance to help teams identify techniques that are more likely to work in their context. This increases the percentage of successful experiments and thereby increases your overall rate of process improvement.

We’re often asked by people when it makes sense for a team to take a traditional approach to software development, and more generally when it makes sense on any type of project. While the focus of this article is on software development projects it also discusses non-software projects too.

Sometimes people are honestly trying to identify when each potential lifecycle, including the traditional lifecycle, makes sense to follow. In many cases this request is coming from people who are desperate to continue working in the same old way that they’re comfortable with. They often hope that the context of the situation that they’re in puts them in a position where they don’t need to adopt new ways of working. Some people get “lucky”, although how lucky it is to forgo an opportunity to gain new skills that are currently in demand is questionable at best, but most find that they need to join their colleagues in adopting agile.

Traditional development was prevalent in the 70s through 90s, but starting in the mid-90s organizations started to adopt iterative approaches such as the Unified Process (UP) as well as agile approaches such as Scrum or Extreme Programming (XP). Having said all this, the majority of organizations still have a few teams following traditional lifecycles and will often have people who believe that traditional software development is a good idea. And, as I argue in this blog, in a few situations they’re right about that.

This blog explores several issues surrounding the traditional approach:

1. What is traditional development?
2. When does it make sense to take a traditional approach?
3. What factors have no bearing on when traditional makes sense?
4. But isn’t traditional better at scale?
5. Does Disciplined Agile (DA) support a traditional lifecycle?
6. Doesn’t Disciplined Agile Delivery (DAD) have a serial lifecycle?
7. What about non-software projects?
8. Why should you listen to me?

1. What is Traditional Development?

Traditional development - also called serial, waterfall, linear, or "predictive" - is based on the idea that the delivery lifecycle should be organized into phases or stages. Figure 1 depicts a classic waterfall lifecycle, similar to what Winston Royce originally proposed (and recommended against) in his 1970 paper. In a pure waterfall the flow is unidirectional (from Requirements to Architecture to…) whereas Figure 1 depicts a “modified waterfall” using bi-directional flow that indicates feedback from a phase may go back to the previous phase.

Figure 1. The Waterfall lifecycle.

Figure 2 depicts a gated waterfall where you need to pass through a “quality gate” to move between phases. The “quality gate” tends to be based on the review and acceptance of artifacts – for example a Software Requirements Specification (SRS) coming out of Requirements, a Software Architecture Design (SAD) coming out of Architecture, and so on. It also shows how feedback can go back to any phase via a Change Control process, which on most traditional teams proves to be a change prevention process in practice. I used quotes around “quality gate” because “quality gates” tend to have very little to do with quality in practice and have a lot to do with promoting questionable bureaucracy. My point is that traditional rhetoric may be blinding you to serious deficiencies in this approach.

Figure 2. The Gated Waterfall lifecycle.

Figure 3 depicts the V-model version of the traditional approach where the test phase is depicted as multiple stages and there are clear indications of what each testing activity validates (for example, Integration Test validates your architecture).

Figure 3. The V-model lifecycle.

The governance strategy with a traditional approach tends to be artifact based (i.e. someone reviews and signs off on your architecture document) rather than risk based (i.e. we proved that the architecture works by building a vertical slice of the solution that implements high-risk requirements). Each phase tends to be staffed with specialists (i.e. requirements are captured by requirements analysts, the design is created by designers, and so on) which tends to motivate a quality gate approach, the development and maintenance of traceability information, and significant management effort to coordinate everything. This in turn increases cost, overall cycle time and decreases ability to support changing requirements (traditionalists typically fear “scope creep” whereas agilists embrace evolving requirements) which in turn decreases stakeholder satisfaction with the end product.

2. When Does Traditional/Waterfall Make Sense?

There are several situations when the traditional approach makes sense:

1. Your team has a traditional culture and skillset. Some teams haven’t adopted an agile mindset nor gained agile skills yet, for whatever reason. Teams are more likely to succeed following a strategy that they understand as opposed to one that they don’t.
2. The project is low risk. The serial nature of the traditional lifecycle makes it poorly suited to address risk, regardless of what adherents of traditional claim. It’s interesting to note that in the original whitepaper describing the traditional lifecycle that Winston Royce pointed out that it wasn’t appropriate for high-risk endeavors, an important tidbit of advice that many organizations have unfortunately ignored over the decades.
3. You’ve done this sort of thing before. One category of project where traditional seems to thrive are “repeat projects” – you’ve done it before and you know what you’re doing so it’s unlikely that you’ll run into unpleasant surprises. An example of this type of project is the installation of a software package by a company that specializes in doing exactly this. This in effect is a type of low-risk project.
4. The requirements are unlikely to evolve. There are some projects where the requirements typically don’t change, such as the deployment of a new version of an operating system across many machines (think Windows upgrade) or a straightforward regulatory project (note that some regulatory requirements do evolve in practice, in particular regulations that are “politically charged”). This is also a type of low-risk project.
5. Transparency isn’t important. Regardless of all the documentation that is generated by a traditional team, the fact is that it’s difficult for senior management to ensure that they have true insight into what is happening on traditional project teams. Another way of saying this is that it’s relatively easy for traditional project managers to mislead you with fanciful status reports and for team members to generate documentation that they believe reviewers are looking for. This lack of transparency is one of the many reasons why traditional software teams have a lower success rate on average than agile teams – traditional teams will be in trouble and management won’t know and therefore be unable to guide them out of trouble early on when it is still easy and inexpensive to do so.

3. What Factors Have No Bearing on When Traditional Makes Sense?

My experience is that the following issues do not have an impact, or when they do have very little impact, on the deciding if a traditional strategy is appropriate for your team:

1. The domain you’re working in. I don’t know of a single domain where agile and lean techniques are not being successfully applied in practice. Not one. We often run into people that say “Sure, agile is fine for e-commerce, but we work in bank so we need to be traditional” or “Sure, agile is fine for banks, but we work in a retailer so we need to be traditional” or “Sure, agile is fine for retailers, but we work for an automobile manufacturer so we need to be traditional” or “Sure, agile is fine for automobile companies but we work for an e-commerce company so we need to be traditional.” But please, don’t take my word for this. Instead do a quick web search on “Agile software development in X” where X is the domain that you think requires a traditional approach – you’ll soon find that many people have figured out how to move beyond traditional in that domain.
2. The desire to “get it right” the first time. When getting it right the first time the quality focused, tight feedback cycle approaches taken by Disciplined Agilists are far more effective in practice than the heavyweight traditional approaches. When someone tells you that they require a traditional approach to get it right, chances are exceptionally good they know every little about enterprise-class agile development strategies such as DA.
3. A desire for “predictability.” Agile approaches, due to their greater transparency, are far more predictable than traditional approaches. The heavy documentation and “quality gates” of traditional provide a false sense of predictability to stakeholders who then get frustrated when traditionalists go over budget, are late, drop significant scope to come in on-time and on-budget, or force stakeholders to accept something built to specification instead of something that meets their real needs. You may find the article Examining the Big Requirements Up Front (BRUF) Approach to be an enlightening read.
4. You’re working at scale. See next section.

4. But Isn’t Traditional Better at Scale?

Not really. Agile and lean are as good or better when applied by large teams, geographically distributed teams, teams in regulatory situations, teams facing domain complexity, teams facing technical complexity, and teams facing organizational distribution. Figure 4 depicts a radar chart of potential tactical scaling factors.

Figure 4. Scaling factors faced by software development teams.

So how does traditional fair in comparison with agile and lean strategies in practice? If you poke around at the IT Surveys page you’ll see that the 2014 Software Development at Scale study found that agile teams outperformed non-agile teams across a range of success factors. Similar results can also be found in the 2006, 2008, 2010, 2011, and 2013 Project Success Surveys. The Dr. Dobb’s Journal 2008 Project Success study found that when comparing agile and traditional approaches based on level of geographic distribution that agile teams were at least as successful and often more so, on average, than traditional teams. In other words, agile was less risky. In 2006 DDJ found that agile was as good or better than traditional regardless of team size, but unfortunately I can no longer find those results online.

To be clear, some of these surveys are a bit long in the tooth. Having said that, in the past few years organizations have figured out how to successfully apply agile and lean approaches at scale.   I suspect that agile and lean will both have pulled ahead even further compared with traditional at scale. We’re actively looking into these sorts of issues so stayed tuned to this blog for future research results. And, we’re not the only ones who are getting these sorts of results. Go poking around on the web and find out for yourself.

5. Does Disciplined Agile (DA) Support the Traditional Lifecycle?

The DA toolkit does not yet support a traditional lifecycle, although an upcoming release in Q2 2020 will bring some aspects of serial/traditional into play.

Having said that, DA has always recognized that in the vast majority of organizations you will have a multi-modal approach where some teams are following an agile approach, some are lean, some are taking a continuous delivery approach, and some are still following traditional. The more disciplined your organization, the more skilled your people, the less likely it is to have traditional teams in practice.

6. Doesn’t DA Support a Serial Lifecycle?

Yes, but it’s risk-based, not artifact nor activity based as you see with traditional approaches. The two project lifecycles supported by DAD, the Scrum-based Agile lifecycle and the Kanban-based Lean lifecycle, have three phases: Inception, Construction, and Transition. These phases, overviewed in Figure 5, capture the ideas that a project team needs to be initiated somehow (Inception), the solution must be developed (Construction), and then released into Production (Transition).

Figure 5. The phases and milestones of DAD project teams.

When you have stable (long-lived) product teams Inception tends to disappear (with the exception of the need for occasional modeling and planning sessions to think things through) and Transition evolves from a multi-day or multi-week phase into a multi-minute or multi-hour activity. Furthermore, the lifecycle evolves over time as you improve, moving from a phased agile/lean project lifecycle into a continuous delivery lifecycle. More on this in future blogs.

7. What About Non-Software Projects?

Although much of the advice presented above applies to non-software projects there are some differences in the non-software world.  For example, in the physical construction world (think building buildings, oil refineries, highways, ...) they will often work in a traditional manner.  This happens for many good reasons:

1. There is significant experience and culture around working this way in those domains.
2. The requirements do not evolve in dramatic ways, unlike with software. 
3. The building materials are known, and in some cases have been available for thousands of years
4. The building materials are inflexible, so changes later on (e.g. the bridge needs to be moved 50m downstream) are incredibly expensive.

Having said that, even when it makes sense to follow a more traditional approach you can still benefit from applying agile strategies on your project.  For example I was recently involved in a physical construction project and I was struck by how much of the actual work was performed in an agile manner.  At a high-level it looked like a traditional/serial project, and it was, but on a day-to-day basis it was reasonably agile within the limits they had.  So it was more of a hybrid strategy masquerading as traditional.

8. Why Should You Listen to Me?

Here is a brief description of my background in traditional software development:

1. I spent the late 80s to mid-90s doing traditional development. In fact, two of my books, Process Patterns and More Process Patterns that I wrote in the mid-1990s, describe a CMMI-compliant traditional approach to software development using object-oriented technologies. These books captured my experiences on a range of traditional and quasi-traditional software development projects.
2. I have significant Unified Process (UP) experience. I have many years of experience with teams following the Unified Process, albeit an iterative methodology rather than traditional one. However, I’ve worked with many organizations that unfortunately instantiated the Unified Process in a traditional manner and then afterwards helped them to recover and evolve their approach into an iterative one and sometimes even an agile one.
3. Disciplined Agile (DA) leverages traditional strategies. In the DA toolkit we purposefully leverage strategies from traditional, iterative, agile, and lean sources in recognition that great ideas come from a variety of sources, including traditional ones. In other words, Disciplined Agile is a hybrid framework.
4. I work with customers still doing traditional today. Almost organization that I work with is multi-modal, with some agile teams, some lean teams, some continuous delivery teams, and yes, even some traditional teams. Most of these organizations are looking to improve their overall productivity, which typically means they adopt even more agile and lean strategies and reduce their overall investment in traditional ways of working.
5. I explicitly research the applicability of various approaches to software development. I have been actively surveying, and sharing all of the results of those surveys publicly in a completely open manner, since the mid-2000s. Several of these research efforts focused on comparing agile, lean, iterative, and traditional approaches to software development in an effort to identify where each approach works well and where they don’t work so well. More on this later as well.

The point is that I believe I have sufficient experience with traditional approaches to speak about where it does and doesn’t work. I hope this blog has provided some valuable insights for you.

Yesterday Mark and I double teamed on a webcast overviewing DAD to a university class.  After the webcast we got the following question:

I am wondering if it would make sense to entirely eliminate the word “phase” from DAD’s vocabulary. What about simply talking about different types of iteration? For instance, inception could become something like “pre-construction iterations”, construction could become “construction iterations”, and transition “transitions iterations”…  That may be a bit cumbersome, but the word phase does scare people away.

Yes, the word phase might scare some people away.  We’ve thought long and hard about the terminology that we use and were also a bit worried about the word “phase” at first.  For some reason phase has become one of those dirty words in the agile community, along with other words such as governance or documentation.  The primary concern seems to be that phase implies a serial approach, something many agilists believe to be foul and evil.  Yet the reality is that projects do in fact proceed in a serial manner.  There are some project initiation activities at the start.  Then there are construction iterations/sprints one after the other in yes, a serial manner.  Then there is an effort to deploy/release your solution into production.  This is clearly “serial in the large”.

But, why the term phase?  Why not iteration?  Or season for that matter season (Gary Evans has a very coherent argument for that term)?  Because phase is accurate (albeit an agile swear word).  In the various IT surveys that I’ve run over the years I’ve found that the average agile team spends about 4.5 weeks performing initiation efforts, has construction iterations that are about 2 weeks in length, and takes an average of roughly 4 weeks to release their solution into production.  So perhaps these initiation and release efforts could each be thought of as two iterations (on average, your mileage may vary) but they really aren’t iterations in their own right usually.  Maybe they’re different length iterations?  There are several ways of thinking about it, but notice how the application of the term iteration doesn’t really make perfect sense.  Then we have numbering issues.  Is the initiation effort iteration/sprint 0?  If it’s two iterations would they be -1 and 0?  Sigh.

One thing that we have done which some may feel to be radical is we’ve chosen to adopt phase names from Unified Process (UP) – Inception, Construction, and Transition.  We could have made up different names, such as Initiation, Development, and Release respectively.  Or adopted Start Up, Construction, and Hardening (yuck, there’s more to Transition than hardening and frankly I would consider a late “hardening” effort to be a clear sign you’ve likely got a process problem you need to deal with).  And I have no doubt that there are many other options for each phase name.  Whatever names we choose someone isn’t going to be happy, and why not give a bit of a nod to a proven software development framework such as UP?

Another interesting thought is that by having named phases it makes it clearer where potential governance points in the lifecycle are.  In the diagram below you can see how several of the suggested light-weight milestones – Stakeholder vision, Sufficient functionality, and Delighted stakeholders – corresponds to the end of a phase.  Or perhaps more accurately, fulfilling the milestone is an indication that your team is moving from one phase to the next.

In the end, I think we’re pretty clear that when you tailor DAD that you can use whatever terminology you like.  In fact one financial institution that adopted an early version of the DAD basic lifecycle, the one based on Scrum, reworked the diagram to use Scrum terminology.  I think it’s a bit goofy but it works for them.

The term phase might in fact scare a few people away.  But, it’s hard to imagine that anyone that concerned about what something is named will be flexible enough to be successful at DAD anyway.