Disciplined Agile

We recently published a collection of articles overviewing the seven principles behind the Disciplined Agile (DA) toolkit.  The article is entitled Principles Behind the Disciplined Agile Framework and it in turn is supported by detailed articles, one for each principle:

1. Delight Customers – We delight our customers when our products and services not only fulfill their needs and expectations but surpass them.
2. Be Awesome – Awesome teams are built around motivated individuals who are given the environment and support required to fulfill their objectives.
3. Pragmatism – Let’s be as effective as we can be, and that may mean we go beyond being just agile.
4. Context Counts – Every person, every team, and every organization is unique.  Let’s find and evolve an effective strategy given the situation we actually face.
5. Choice is Good – Different contexts require different strategies. Teams need to be able to own their own process and to experiment to discover what works in practice for them given the situation that they face. Having process options to choose from, and understanding the trade-offs of those options, enables you to home in on better options sooner.
6. Optimize Flow – Your organization is a complex adaptive system (CAS) of interacting teams and groups that individually evolve continuously and affect each other as they do. To succeed you must ensure that these teams are well aligned, remained well aligned, and better yet improve their alignment over time.
7. Enterprise Awareness – When people are enterprise aware they are motivated to consider the overall needs of their organization, to ensure that what they’re doing contributes positively to the goals of the organization and not just to the sub-optimal goals of their team.

We hope that you find these articles insightful.

This lifecycle is a natural progression from the Agile lifecycle. Teams typically evolve to this lifecycle from the Agile lifecycle, often adopting iteration lengths of one-week or less. The key difference between this and the Agile lifecycle is that the continuous delivery lifecycle results in a release of new functionality at the end of each iteration rather than after a set of iterations. Teams require a mature set of practices around continuous integration and continuous deployment and other Disciplined DevOps strategies. This lifecycle is suitable when:

• Solutions that can be delivered to stakeholders in a frequent and incremental basis
• Work remains relatively stable within an iteration
• Organizations with streamlined deployment practices and procedures
• Projects where getting value into the hands of stakeholders rapidly, before the entire solution is complete, is critical
• Teams have mature DevOps practices in place including continuous integration, continuous deployment, and automated regression testing
• The team is long-lived (stable), working on a series of releases over time

Related Reading

• Full Agile Delivery Lifecycles

Just as your process must be flexible and adaptive, so must your organization. In Reinventing Organizations Frederick Laloux works through the history, and arguably a maturity model, for organizational design. The premise, which is overviewed in the diagram below (you can click on it for a high-res version), is that over time we’re seeing organizations evolve from tribal and often violent structures (Red) through more formalized hierarchical structures (Amber/Orange) to agile approaches (Green/Teal). Today the vast majority of organizations, believed to be 80-90%, are somewhere on the Amber through Green scale.

There are several important observations we’d like to make about Laloux’s organizational maturity scale:

1. For your organization to support agile it should at least be (mostly) Green, with a participative and values-based culture, or better yet Teal with a truly adaptive and aware strategy (as we’ve been preaching throughout this chapter).
2. Your organization will start its improvement journey wherever it currently is on the scale. Laloux’s model provides insights into what your current challenges are likely to be and what potential improvements you should consider making.
3. Teams can still be agile within Orange and Amber organizations, but the organization itself will struggle with agility due to cultural impediments.
4. It is difficult to jump organizational levels – in other words if your organization is currently Amber then you need to move through Orange, then Green, and finally Teal.
5. To move between levels you need the both top-down support from leadership as well as bottom-up support from front-line staff – bottom up, “stealth” improvement efforts will fail on their own when organizational antibodies fight back.

You Want to be At Least Green

Why does your organization need to be at least Green or Teal to become agile? Green organizations support a participative culture style that enables collaboration within and between teams. Green organizations explicitly align people around common values, so injecting any missing agile and lean philosophies often proves to be straightforward. Teal organizations go one step further and bring it all together by explicitly recognizing that they are complex adaptive systems (CASs). This provides an environment where agile teams are able to experiment, learn, collaborate, and most importantly thrive as they find new ways to delight their customers.

Improving Horizontally is Much More Realistic

Laloux himself is very clear about the importance of top-down support if you want to become a Teal organization, or frankly just to move up the hierarchy (say from Red to Amber).  In chapter 3 of Reinventing Organizations he states that for an organization to become Teal that it requires the support of both the CEO/founder and the owners of the company.  Our experience is that a third factor is required, the support of the front-line staff (and better yet middle management), for your transformation to be successful.

Laloux believes that it’s much easier for organizations to improve horizontally – become the best Orange or Amber organization that you can be.  In many ways this is much more conducive to a lean continuous improvement strategy than an “agile transformation” strategy.

Your Organization is Probably a Rainbow

It’s attractive to think that your organizational culture is consistent across the entire enterprise, but it is far more likely that you have teams or divisions with differing color ratings according to Laloux’s model. This is because the culture of a team (or division) is greatly influenced by the leader of that team, and leaders vary in their style, and because teams face unique situations – sometimes a red strategy is the most appropriate given what the team faces. Context counts!

Suggested Reading

• Reinventing Organizations: A Guide to Creating Organizations Inspired by the Next Stage in Human Consciousness by Frederic Laloux (2014)
• Too Many Fluffy Agile Bunnies

We’re often asked by people when it makes sense for a team to take a traditional approach to software development, and more generally when it makes sense on any type of project. While the focus of this article is on software development projects it also discusses non-software projects too.

Sometimes people are honestly trying to identify when each potential lifecycle, including the traditional lifecycle, makes sense to follow. In many cases this request is coming from people who are desperate to continue working in the same old way that they’re comfortable with. They often hope that the context of the situation that they’re in puts them in a position where they don’t need to adopt new ways of working. Some people get “lucky”, although how lucky it is to forgo an opportunity to gain new skills that are currently in demand is questionable at best, but most find that they need to join their colleagues in adopting agile.

Traditional development was prevalent in the 70s through 90s, but starting in the mid-90s organizations started to adopt iterative approaches such as the Unified Process (UP) as well as agile approaches such as Scrum or Extreme Programming (XP). Having said all this, the majority of organizations still have a few teams following traditional lifecycles and will often have people who believe that traditional software development is a good idea. And, as I argue in this blog, in a few situations they’re right about that.

This blog explores several issues surrounding the traditional approach:

1. What is traditional development?
2. When does it make sense to take a traditional approach?
3. What factors have no bearing on when traditional makes sense?
4. But isn’t traditional better at scale?
5. Does Disciplined Agile (DA) support a traditional lifecycle?
6. Doesn’t Disciplined Agile Delivery (DAD) have a serial lifecycle?
7. What about non-software projects?
8. Why should you listen to me?

1. What is Traditional Development?

Traditional development - also called serial, waterfall, linear, or "predictive" - is based on the idea that the delivery lifecycle should be organized into phases or stages. Figure 1 depicts a classic waterfall lifecycle, similar to what Winston Royce originally proposed (and recommended against) in his 1970 paper. In a pure waterfall the flow is unidirectional (from Requirements to Architecture to…) whereas Figure 1 depicts a “modified waterfall” using bi-directional flow that indicates feedback from a phase may go back to the previous phase.

Figure 1. The Waterfall lifecycle.

Figure 2 depicts a gated waterfall where you need to pass through a “quality gate” to move between phases. The “quality gate” tends to be based on the review and acceptance of artifacts – for example a Software Requirements Specification (SRS) coming out of Requirements, a Software Architecture Design (SAD) coming out of Architecture, and so on. It also shows how feedback can go back to any phase via a Change Control process, which on most traditional teams proves to be a change prevention process in practice. I used quotes around “quality gate” because “quality gates” tend to have very little to do with quality in practice and have a lot to do with promoting questionable bureaucracy. My point is that traditional rhetoric may be blinding you to serious deficiencies in this approach.

Figure 2. The Gated Waterfall lifecycle.

Figure 3 depicts the V-model version of the traditional approach where the test phase is depicted as multiple stages and there are clear indications of what each testing activity validates (for example, Integration Test validates your architecture).

Figure 3. The V-model lifecycle.

The governance strategy with a traditional approach tends to be artifact based (i.e. someone reviews and signs off on your architecture document) rather than risk based (i.e. we proved that the architecture works by building a vertical slice of the solution that implements high-risk requirements). Each phase tends to be staffed with specialists (i.e. requirements are captured by requirements analysts, the design is created by designers, and so on) which tends to motivate a quality gate approach, the development and maintenance of traceability information, and significant management effort to coordinate everything. This in turn increases cost, overall cycle time and decreases ability to support changing requirements (traditionalists typically fear “scope creep” whereas agilists embrace evolving requirements) which in turn decreases stakeholder satisfaction with the end product.

2. When Does Traditional/Waterfall Make Sense?

There are several situations when the traditional approach makes sense:

1. Your team has a traditional culture and skillset. Some teams haven’t adopted an agile mindset nor gained agile skills yet, for whatever reason. Teams are more likely to succeed following a strategy that they understand as opposed to one that they don’t.
2. The project is low risk. The serial nature of the traditional lifecycle makes it poorly suited to address risk, regardless of what adherents of traditional claim. It’s interesting to note that in the original whitepaper describing the traditional lifecycle that Winston Royce pointed out that it wasn’t appropriate for high-risk endeavors, an important tidbit of advice that many organizations have unfortunately ignored over the decades.
3. You’ve done this sort of thing before. One category of project where traditional seems to thrive are “repeat projects” – you’ve done it before and you know what you’re doing so it’s unlikely that you’ll run into unpleasant surprises. An example of this type of project is the installation of a software package by a company that specializes in doing exactly this. This in effect is a type of low-risk project.
4. The requirements are unlikely to evolve. There are some projects where the requirements typically don’t change, such as the deployment of a new version of an operating system across many machines (think Windows upgrade) or a straightforward regulatory project (note that some regulatory requirements do evolve in practice, in particular regulations that are “politically charged”). This is also a type of low-risk project.
5. Transparency isn’t important. Regardless of all the documentation that is generated by a traditional team, the fact is that it’s difficult for senior management to ensure that they have true insight into what is happening on traditional project teams. Another way of saying this is that it’s relatively easy for traditional project managers to mislead you with fanciful status reports and for team members to generate documentation that they believe reviewers are looking for. This lack of transparency is one of the many reasons why traditional software teams have a lower success rate on average than agile teams – traditional teams will be in trouble and management won’t know and therefore be unable to guide them out of trouble early on when it is still easy and inexpensive to do so.

3. What Factors Have No Bearing on When Traditional Makes Sense?

My experience is that the following issues do not have an impact, or when they do have very little impact, on the deciding if a traditional strategy is appropriate for your team:

1. The domain you’re working in. I don’t know of a single domain where agile and lean techniques are not being successfully applied in practice. Not one. We often run into people that say “Sure, agile is fine for e-commerce, but we work in bank so we need to be traditional” or “Sure, agile is fine for banks, but we work in a retailer so we need to be traditional” or “Sure, agile is fine for retailers, but we work for an automobile manufacturer so we need to be traditional” or “Sure, agile is fine for automobile companies but we work for an e-commerce company so we need to be traditional.” But please, don’t take my word for this. Instead do a quick web search on “Agile software development in X” where X is the domain that you think requires a traditional approach – you’ll soon find that many people have figured out how to move beyond traditional in that domain.
2. The desire to “get it right” the first time. When getting it right the first time the quality focused, tight feedback cycle approaches taken by Disciplined Agilists are far more effective in practice than the heavyweight traditional approaches. When someone tells you that they require a traditional approach to get it right, chances are exceptionally good they know every little about enterprise-class agile development strategies such as DA.
3. A desire for “predictability.” Agile approaches, due to their greater transparency, are far more predictable than traditional approaches. The heavy documentation and “quality gates” of traditional provide a false sense of predictability to stakeholders who then get frustrated when traditionalists go over budget, are late, drop significant scope to come in on-time and on-budget, or force stakeholders to accept something built to specification instead of something that meets their real needs. You may find the article Examining the Big Requirements Up Front (BRUF) Approach to be an enlightening read.
4. You’re working at scale. See next section.

4. But Isn’t Traditional Better at Scale?

Not really. Agile and lean are as good or better when applied by large teams, geographically distributed teams, teams in regulatory situations, teams facing domain complexity, teams facing technical complexity, and teams facing organizational distribution. Figure 4 depicts a radar chart of potential tactical scaling factors.

Figure 4. Scaling factors faced by software development teams.

So how does traditional fair in comparison with agile and lean strategies in practice? If you poke around at the IT Surveys page you’ll see that the 2014 Software Development at Scale study found that agile teams outperformed non-agile teams across a range of success factors. Similar results can also be found in the 2006, 2008, 2010, 2011, and 2013 Project Success Surveys. The Dr. Dobb’s Journal 2008 Project Success study found that when comparing agile and traditional approaches based on level of geographic distribution that agile teams were at least as successful and often more so, on average, than traditional teams. In other words, agile was less risky. In 2006 DDJ found that agile was as good or better than traditional regardless of team size, but unfortunately I can no longer find those results online.

To be clear, some of these surveys are a bit long in the tooth. Having said that, in the past few years organizations have figured out how to successfully apply agile and lean approaches at scale.   I suspect that agile and lean will both have pulled ahead even further compared with traditional at scale. We’re actively looking into these sorts of issues so stayed tuned to this blog for future research results. And, we’re not the only ones who are getting these sorts of results. Go poking around on the web and find out for yourself.

5. Does Disciplined Agile (DA) Support the Traditional Lifecycle?

The DA toolkit does not yet support a traditional lifecycle, although an upcoming release in Q2 2020 will bring some aspects of serial/traditional into play.

Having said that, DA has always recognized that in the vast majority of organizations you will have a multi-modal approach where some teams are following an agile approach, some are lean, some are taking a continuous delivery approach, and some are still following traditional. The more disciplined your organization, the more skilled your people, the less likely it is to have traditional teams in practice.

6. Doesn’t DA Support a Serial Lifecycle?

Yes, but it’s risk-based, not artifact nor activity based as you see with traditional approaches. The two project lifecycles supported by DAD, the Scrum-based Agile lifecycle and the Kanban-based Lean lifecycle, have three phases: Inception, Construction, and Transition. These phases, overviewed in Figure 5, capture the ideas that a project team needs to be initiated somehow (Inception), the solution must be developed (Construction), and then released into Production (Transition).

Figure 5. The phases and milestones of DAD project teams.

When you have stable (long-lived) product teams Inception tends to disappear (with the exception of the need for occasional modeling and planning sessions to think things through) and Transition evolves from a multi-day or multi-week phase into a multi-minute or multi-hour activity. Furthermore, the lifecycle evolves over time as you improve, moving from a phased agile/lean project lifecycle into a continuous delivery lifecycle. More on this in future blogs.

7. What About Non-Software Projects?

Although much of the advice presented above applies to non-software projects there are some differences in the non-software world.  For example, in the physical construction world (think building buildings, oil refineries, highways, ...) they will often work in a traditional manner.  This happens for many good reasons:

1. There is significant experience and culture around working this way in those domains.
2. The requirements do not evolve in dramatic ways, unlike with software. 
3. The building materials are known, and in some cases have been available for thousands of years
4. The building materials are inflexible, so changes later on (e.g. the bridge needs to be moved 50m downstream) are incredibly expensive.

Having said that, even when it makes sense to follow a more traditional approach you can still benefit from applying agile strategies on your project.  For example I was recently involved in a physical construction project and I was struck by how much of the actual work was performed in an agile manner.  At a high-level it looked like a traditional/serial project, and it was, but on a day-to-day basis it was reasonably agile within the limits they had.  So it was more of a hybrid strategy masquerading as traditional.

8. Why Should You Listen to Me?

Here is a brief description of my background in traditional software development:

1. I spent the late 80s to mid-90s doing traditional development. In fact, two of my books, Process Patterns and More Process Patterns that I wrote in the mid-1990s, describe a CMMI-compliant traditional approach to software development using object-oriented technologies. These books captured my experiences on a range of traditional and quasi-traditional software development projects.
2. I have significant Unified Process (UP) experience. I have many years of experience with teams following the Unified Process, albeit an iterative methodology rather than traditional one. However, I’ve worked with many organizations that unfortunately instantiated the Unified Process in a traditional manner and then afterwards helped them to recover and evolve their approach into an iterative one and sometimes even an agile one.
3. Disciplined Agile (DA) leverages traditional strategies. In the DA toolkit we purposefully leverage strategies from traditional, iterative, agile, and lean sources in recognition that great ideas come from a variety of sources, including traditional ones. In other words, Disciplined Agile is a hybrid framework.
4. I work with customers still doing traditional today. Almost organization that I work with is multi-modal, with some agile teams, some lean teams, some continuous delivery teams, and yes, even some traditional teams. Most of these organizations are looking to improve their overall productivity, which typically means they adopt even more agile and lean strategies and reduce their overall investment in traditional ways of working.
5. I explicitly research the applicability of various approaches to software development. I have been actively surveying, and sharing all of the results of those surveys publicly in a completely open manner, since the mid-2000s. Several of these research efforts focused on comparing agile, lean, iterative, and traditional approaches to software development in an effort to identify where each approach works well and where they don’t work so well. More on this later as well.

The point is that I believe I have sufficient experience with traditional approaches to speak about where it does and doesn’t work. I hope this blog has provided some valuable insights for you.

A common question that people ask is how does the adoption of agile within a team affect its productivity?  The answer to this question will vary by team, but there are several common patterns that we’ve seen over time.  In this blog we explore:

1. What does increased productivity mean to you?
2. Agile adoption patterns
3. What milestones to look out for as a team adopts agile
4. What you can do to increase your chance of success
5. How do you know productivity improved?
6. Agile is about more than productivity improvement

What Does Increased Productivity Mean to You?

Productivity is defines various measures of the efficiency of production, and is calculated as

Value of Output divided by Cost of Input

The implication of this calculation is that there is flexibility in the way that we can increase the productivity of a team:

• Produce the same output with less cost (i.e. with fewer people).
• Create greater value with the same number of people.
• Deliver value incrementally more often, thereby earning value sooner for a longer period of time (this is called decreasing the cost of delay)

Remember that context counts – each team will choose the most appropriate way for them to increase their productivity. Having said that, a common result of a team adopting agile is to incrementally deliver value more often.

Agile Adoption Patterns

The following diagram overviews three common patterns when it comes to productivity change when teams adopt agile.  You’ve likely seen simpler versions of this diagram elsewhere that only show the dark green line, but our experience is that’s only part of the story. You can see in all three cases that the adoption of agile results in an initial productivity loss on a team – this reflects the reality that with any type of change it will take time for a team to learn the new strategy, to identify how it fits into their environment, and to learn the new requisite skills and behaviours.

The three patterns, from least desirable to most desirable, are:

1. A failed agile adoption (red line). Teams fail to adopt agile for several reasons, usually because the team doesn’t want to adopt agile ways of working, the organization doesn’t properly support their adoption efforts, or the rest of the organization continues to drag them down with traditional ways of working.
2. A successful agile adoption (green line). Luckily most teams succeed in their agile adoption efforts, and numerous studies have shown a wide range of benefits including faster time to delivery, increased quality, increased stakeholder satisfaction with the delivered solution, and improve team morale to name a few.  Every team is different, but overall on average adopting agile is a positive experience.  You’ll land on this curve when you treat agile adoption as a project, something you do for a few months to help make the team more effective, if you’re successful.
3. A successful agile adoption evolving into continuous improvement (green dashed line). The most successful teams realize that process improvement isn’t a short-term project but instead is a long-term journey that you undertake.  This is reflected in the dashed line in the diagram below.  You typically start by following a transformation strategy with the team – you get them some initial training, some coaching, help them change their work environment and tooling to be more collaborative.  Then at some point an improvement mindset begins to take hold within the team, one of the fundamental aspects of agility.  The team reflects on a regular basis, identifying potential ways that they can improve and then they experiment with those strategies to see which ones work in practice for them. It’s at this point that they’ve shifted out of the transformation strategy into more of a continuous improvement strategy, which is what enables them to reach higher levels of productivity than what is typically achieved with just a transformation strategy.

What Are the Key Milestones to Look For?

There are three key milestone points on the successful paths that you should watch out for:

1. Productivity trough (4-8 weeks). With anything new there is always a learning curve, and agile is no exception.  When a team begins to move towards agile their productivity will drop for several reasons: they will likely invest some time taking training, it will take people time to learn new techniques and adopt new ways of working, and it will take time for the team to determine how they will work together following these new agile strategies.  Your productivity levels tend to bottom out after four to eight weeks and then after that will start to improve.  The amount of time varies by team, depending on whether any team members have previous agile experience that they can leverage, how much team members want to change, how effective the training is, and whether you have the support of an experienced and effective coach.
2. Productivity recovers (2-4 months).  For most teams, the ones who are successful at becoming agile, their productivity levels will recover back to the level they were before they started on their agile improvement journey, within two to four months of starting.  This amount of time depends on the same issues mentioned before.
3. Improvement culture takes hold (3-6 months). This is the point where the improvement mindset really kicks in and the team starts to explicitly work together to improve the way that they work. This is a reflection that the team is actually “being agile” and not just doing agile ceremonies. Sadly not all teams reach this point and move up onto the dashed green line in the diagram above.  Whether they do so or not is primarily dependent on the willingness of team members to become agile, the quality of the coaching that they receive, and whether your organizational environment allows them to own their process.

How Can You Improve Your Chance of Success?

There are several strategies that you can employ to increase your chance of successfully adopting agile and shifting to a continuous improvement culture within your team:

1. Invest in training. Get the team started on the right foot with training that not only goes into the fundamental concepts behind agile (“being agile” training) but also works through from beginning to end how agile works in practice (“doing agile” training).  Being agile training is incredibly easy to find, but good “doing agile” training that is comprehensive is much harder to find.  Luckily there are several very good Disciplined Agile training offerings that focus on how to “do agile” in enterprise-class settings.
2. Hire an experienced coach. A good coach will help your team to avoid common learning pitfalls, and better yet quickly guide you through “learning experiences”, working through with you how to improve the way the team works. Hiring a coach can be a challenge because as we show in Why is it so hard to find qualified coaches? it is possible for anyone, and unfortunately they often do, to claim that they are an agile coach.  Effective coaches have deep experience in what they are coaching as well as skills in the act of coaching itself.  The majority of “agile coaches” tend to to be short on both of these things, and the few coaches that are qualified are in high demand.  There are several Certified Disciplined Agile Partners that you may want to reach out to for Certified Disciplined Agile Coaches (CDACs).
3. Give the team an “organizational pass.” It’s incredibly difficult for a team to become agile when they are still surrounded by other groups that are actively working in a non-agile manner.  Agile teams need to collaborate with others to achieve their goals (in fact, the 2016 Agility at Scale study found that 96% of people indicated that their team needs to interact with at least one other team).  So, if you want to enable a team to become more agile and improve then you also need to motivate these other teams they rely on – such as the data team, the enterprise architects, and even the governance team – to be sufficiently flexible to work with the agile team in an agile manner.  In some cases the agile team may need to be “given a pass” from creating the mandated artifacts, or having to jump through the mandated “quality gates”, required by these teams.
4. Help the teams that they collaborate with to become more agile.  The next step after giving an agile team an organizational pass it to recognize that this is an opportunity to experiment with improving other areas within your organization.  Help the enterprise architects to learn about agile and to try a few agile strategies themselves.  Similarly the data team can experiment with agile data management strategies and certainly your IT governance team can also take the opportunity to up their game as well.

How Do You Know That Your Productivity Actually Improved?

Although the chart above intuitively makes sense, how do you know that your productivity has actually increased?  To definitively answer this question you need to determine what productivity means to you, what the productivity level of the team currently is, and then continue to measure the level of productivity over time. This strategy tends to fall apart because few organizations know how they want to measure team productivity and fewer yet have actual measures in place. This of course is particularly vexing when senior management still requires you to prove that your productivity has increased as the result of your agile adoption efforts.  Luckily there are ways to measure the change in productivity even when you don’t know what the baseline productivity level currently is.  We’ll address this topic in a future blog.

Agile is About More Than Productivity Improvement

There are many benefits to agile, improving team productivity being just one of them. Potential benefits, some of which lead to greater productivity, include:

• Improving the quality of the delivered solution
• Improved stakeholder satisfaction
• Greater adaptability to market changes
• Increased team morale
• Quicker time to market
• Greater frequency of delivery
• and many more