Disciplined Agile

At the Agile 2017 conference this week Lynn Winterboer was kind enough to invite me to her workshop which explored how to apply agile strategies in the data space.  She did a great job of facilitating the group of about 40 people through identifying the challenges currently faced by teams. The main issues that the group explored were:

• Product ownership and data
• Agile data architecture
• Data testing and tooling support
• How to include data people, and activities, in development

Lynn will soon be blogging about the results so I’m not going to dive into that here.  I suspect that her blog post will be very interesting.

What I’d like to do here is share a few thoughts about what I observed:

1. The discussion was very healthy. This was a group of very smart people coming from different backgrounds.  Everyone was interested in sharing their experiences and working together to solve some tough problems.
2. Context counts. “Agile and data” is a big topic.  A few people were dealing with the issue of how to address data issues on application development projects, some were focused on agile data warehousing/business intelligence, and some were focused on complex data analytics.  In our conversations it was very clear that strategies which worked for app development wouldn’t work for analytics, and vice versa.  This is why Context Counts is one of Disciplined Agile’s fundamental principles.
3. The challenges are tough. Everyone was working in organizations that were struggling with these challenges.  For each of issue we could have spent much longer exploring the potential solution(s).
4. Every challenge we discussed are solved issues.  I’ve always found it frustrating when people, who are very smart and who have been struggling with a problem for awhile, have clearly never thought to simply google “database testing tools” or “agile architecture” to find out what advice is already out there.  When we discussed testing I even asked why people hadn’t done such as search, and then pointed out that there are a lot of tools available and even a few people maintaining lists of such tools (such as 40+ database testing tools).  All of these challenges, and more, have solutions described by techniques of the Agile Data and Agile Modeling methods from about 15 years ago.  These techniques have of course been adopted, and put into context, by the Disciplined Agile (DA) toolkit and in particular Disciplined Agile Delivery (DAD).
5. The “factions” behaved differently.  Long ago I pointed out that there’s a cultural impedance mismatch between the data and developer communities, and it was pretty easy to observe during this workshop.  For example, during the workshop we were asked to identify solutions to the challenges.  Lynn wanted us to write this information on flip-chart paper so that she could later scan it and share it with others.  For awhile the groups dominated by data people discussed the solutions without writing anything down.  Their conversations were good, but they quickly got stuck in analysis paralysis mode because they seemed to be afraid to write down information for fear that they couldn’t easily update it (the challenge with having paper to write on instead of whiteboards). The groups dominated by agile people, the ones focused on development and architecture, started by writing on sticky notes and putting them on the flip chart paper.  This fundamental agile modelling strategy enabled them to visualize and share their work transparently, enhancing their conversation and enabling them to move forward easily.
6. Database (tool) vendors need to up their game. Speaking out tools, database vendors and data warehouse tool vendors really need to up their game.  Here’s a few very harsh questions: Does your database tool vendor or ETL tool vendor offer testing tools that enable both black box and white box database testing?  Answer: very likely no, because their customers don’t demand that of them.  Did your data team even think to ask for such tools?  Answer: very likely not.  How many database testing books do you think have been written? Answer: very few, go do a search and see for yourself.  Why does the data community have such a huge blind spot when it comes to testing? Answer: this is a huge side effect of the cultural impedance mismatch.

I’m very happy to see that Lynn is actively trying to bridge the agile and data communities, helping us to learn from each other.  This is something she’s been doing for years and doing it quite well.  My experience is that both communities would benefit greatly from more collaboration with each other.

As you know the Disciplined Agile (DA) toolkit supports several delivery lifecycles – Agile, Lean, Continuous Delivery: Agile, Continuous Delivery: Lean, Exploratory, and Program.  We do this because solution delivery teams face different situations, so one lifecycle will not fit all.  This begs the question of when would a team adopt each lifecycle?  That is the focus of this blog.

This blog explores the following topics:

1. What lifecycles does DAD support?
2. What criteria should you consider when choosing a lifecycle?
3. Who chooses?

What Lifecycles Does DAD Support?

The delivery lifecycles we will explore are:

1. Agile. DAD’s Agile lifecycle which extends Scrum’s construction lifecycle. Common scenarios for adopting this version of the lifecycle include situations where you’re extending Scrum to be sufficient for your needs or you’re taking an agile project approach.
2. Advanced. The Lean lifecycle encompasses lean strategies such as minimizing work in process, maximizing flow, a continuous stream of work (instead of fixed iterations), and reducing bottlenecks. New work is pulled from the work item pool when the team has capacity.
3. Continuous Delivery: Agile.  The Continuous Delivery: Agile lifecycle  is a natural progression from the Agile lifecycle. Teams typically evolve to this lifecycle from the Agile/Basic lifecycle, often adopting iteration lengths of one-week or less. The key difference between this and the Agile lifecycle is that the continuous delivery lifecycle results in a release of new functionality at the end of each iteration rather than after a set of iterations.
4. Continuous Delivery: Lean. DAD’s Continuous Delivery: Lean lifecycle is basically a leaner version of the Continuous Delivery: Agile lifecycle where the product is shipped into production or the marketplace on a very regular basis. This could be often as daily, although weekly or monthly is quite common too.
5. Exploratory/Lean Startup.  DAD’s Exploratory lifecycle is followed by agile or lean teams that find themselves in startup or research situations where their stakeholders have an idea for a new product but they do yet understand what is actually needed by their user base.
6. Program. DAD’s Program lifecycle describes how to coordinate a team of teams.  It is very similar to the LeSS lifecycle although does not prescribe that the subteams/squads need to be following Scrum.  It can also be thought of as a simplified version of the SAFe lifecycle.
7. Waterfall/Serial.  Although this is not supported by Disciplined Agile, we do recognize that some teams will still follow this more traditional way of working.  For more thoughts on this subject, please see the blog posting When Does Traditional Software Development Make Sense?

What Criteria Should You Consider When Choosing a Lifecycle?

The following table compares the lifecycles, suggesting when you would choose to follow each one.

Who Should Choose the Lifecycle?

The team.

They will often do this with the guidance of an experienced Disciplined Agile coach, particularly when they are new to DA.  It’s tempting to have your portfolio management team to make this choice, and they may often make a (hopefully solid) suggestion when the first initiated an endeavour, but in the end it needs to be the choice of the team.  As you see in the table above there are common considerations for when to use each lifecycle, but the primary considerations are always the skill and preferences of the team itself.

Related Reading

• Evolve WoW process goal
• Full Delivery Lifecycles
• Principle: Choice is Good
• Principle: Context Counts
• Principle: Optimize the Flow

We recently published a collection of articles overviewing the seven principles behind the Disciplined Agile (DA) toolkit.  The article is entitled Principles Behind the Disciplined Agile Framework and it in turn is supported by detailed articles, one for each principle:

1. Delight Customers – We delight our customers when our products and services not only fulfill their needs and expectations but surpass them.
2. Be Awesome – Awesome teams are built around motivated individuals who are given the environment and support required to fulfill their objectives.
3. Pragmatism – Let’s be as effective as we can be, and that may mean we go beyond being just agile.
4. Context Counts – Every person, every team, and every organization is unique.  Let’s find and evolve an effective strategy given the situation we actually face.
5. Choice is Good – Different contexts require different strategies. Teams need to be able to own their own process and to experiment to discover what works in practice for them given the situation that they face. Having process options to choose from, and understanding the trade-offs of those options, enables you to home in on better options sooner.
6. Optimize Flow – Your organization is a complex adaptive system (CAS) of interacting teams and groups that individually evolve continuously and affect each other as they do. To succeed you must ensure that these teams are well aligned, remained well aligned, and better yet improve their alignment over time.
7. Enterprise Awareness – When people are enterprise aware they are motivated to consider the overall needs of their organization, to ensure that what they’re doing contributes positively to the goals of the organization and not just to the sub-optimal goals of their team.

We hope that you find these articles insightful.

This lifecycle is a natural progression from the Agile lifecycle. Teams typically evolve to this lifecycle from the Agile lifecycle, often adopting iteration lengths of one-week or less. The key difference between this and the Agile lifecycle is that the continuous delivery lifecycle results in a release of new functionality at the end of each iteration rather than after a set of iterations. Teams require a mature set of practices around continuous integration and continuous deployment and other Disciplined DevOps strategies. This lifecycle is suitable when:

• Solutions that can be delivered to stakeholders in a frequent and incremental basis
• Work remains relatively stable within an iteration
• Organizations with streamlined deployment practices and procedures
• Projects where getting value into the hands of stakeholders rapidly, before the entire solution is complete, is critical
• Teams have mature DevOps practices in place including continuous integration, continuous deployment, and automated regression testing
• The team is long-lived (stable), working on a series of releases over time

Related Reading

• Full Agile Delivery Lifecycles

Just as your process must be flexible and adaptive, so must your organization. In Reinventing Organizations Frederick Laloux works through the history, and arguably a maturity model, for organizational design. The premise, which is overviewed in the diagram below (you can click on it for a high-res version), is that over time we’re seeing organizations evolve from tribal and often violent structures (Red) through more formalized hierarchical structures (Amber/Orange) to agile approaches (Green/Teal). Today the vast majority of organizations, believed to be 80-90%, are somewhere on the Amber through Green scale.

There are several important observations we’d like to make about Laloux’s organizational maturity scale:

1. For your organization to support agile it should at least be (mostly) Green, with a participative and values-based culture, or better yet Teal with a truly adaptive and aware strategy (as we’ve been preaching throughout this chapter).
2. Your organization will start its improvement journey wherever it currently is on the scale. Laloux’s model provides insights into what your current challenges are likely to be and what potential improvements you should consider making.
3. Teams can still be agile within Orange and Amber organizations, but the organization itself will struggle with agility due to cultural impediments.
4. It is difficult to jump organizational levels – in other words if your organization is currently Amber then you need to move through Orange, then Green, and finally Teal.
5. To move between levels you need the both top-down support from leadership as well as bottom-up support from front-line staff – bottom up, “stealth” improvement efforts will fail on their own when organizational antibodies fight back.

You Want to be At Least Green

Why does your organization need to be at least Green or Teal to become agile? Green organizations support a participative culture style that enables collaboration within and between teams. Green organizations explicitly align people around common values, so injecting any missing agile and lean philosophies often proves to be straightforward. Teal organizations go one step further and bring it all together by explicitly recognizing that they are complex adaptive systems (CASs). This provides an environment where agile teams are able to experiment, learn, collaborate, and most importantly thrive as they find new ways to delight their customers.

Improving Horizontally is Much More Realistic

Laloux himself is very clear about the importance of top-down support if you want to become a Teal organization, or frankly just to move up the hierarchy (say from Red to Amber).  In chapter 3 of Reinventing Organizations he states that for an organization to become Teal that it requires the support of both the CEO/founder and the owners of the company.  Our experience is that a third factor is required, the support of the front-line staff (and better yet middle management), for your transformation to be successful.

Laloux believes that it’s much easier for organizations to improve horizontally – become the best Orange or Amber organization that you can be.  In many ways this is much more conducive to a lean continuous improvement strategy than an “agile transformation” strategy.

Your Organization is Probably a Rainbow

It’s attractive to think that your organizational culture is consistent across the entire enterprise, but it is far more likely that you have teams or divisions with differing color ratings according to Laloux’s model. This is because the culture of a team (or division) is greatly influenced by the leader of that team, and leaders vary in their style, and because teams face unique situations – sometimes a red strategy is the most appropriate given what the team faces. Context counts!

Suggested Reading

• Reinventing Organizations: A Guide to Creating Organizations Inspired by the Next Stage in Human Consciousness by Frederic Laloux (2014)
• Too Many Fluffy Agile Bunnies