Disciplined Agile

A process blade encompasses a cohesive part of your overall organizational way of working (WoW). Each process blade addresses a specific organizational capability, such as Data Management, Continuous Improvement, or Vendor Management. Process blades are sometimes called process areas, key process areas (KPAs), or business functions. 

In September 2016 InfoQ published an interview, Benefits of Agile Transformation at Barclays, with Jonathon Smart and Ian Dugmore of Barclays in the UK.  The interview summarizes the experiences of adopting Disciplined Agile (DA) strategies within Barclays.  Some of the key points made in the interview include:

• Barclays is a large financial institution with over 130,000 employees that has been in operation for over 325 years
• Barclays is taking a holistic approach to their agile transformation, it is not just a technology thing, linking up their various “islands of agility” to reduce the impedance mismatch between them
• Barclays has the equivalent of over 800 agile teams that have adopted DA approaches in progress
• They considered SAFe, LeSS and DA and settled on DA as it is not a “one-size fits all” approach
• They needed to change both their funding and measurement strategies
• Results include: A 300% increase in throughput (note: measured in terms of story delivery); 50% reduction in code complexity; test code coverage increase by 50%; More than half of teams are deploying into production at least monthly; Improved team happiness/morale; Improved business outcomes such as quicker time to market

The source article is definitely a very interesting read.  

This brief article explains our thinking around our terminology choices in the Disciplined Agile (DA) toolkit. It overviews the terminology principles that we follow, discusses why Scrum terminology isn’t appropriate, and maps common Scrum terms to DA terms.

Our Principles Around Terminology

The following three principles drive our terminology decisions:

1. Terms must be clear. If you need to explain the term, it likely isn’t the best. For example, how many times have you had to explain what a Scrum meeting is? Call it a coordination meeting instead, and people have a much better idea of what’s going on.
2. Terms must be method neutral. Every team is unique and owns its own process. Part of owning your process is choosing the overall method, or lifecycle, that you’re following. Because the DA toolkit is a hybrid that leverages a variety of methods, were we to adopt one method’s terminology over another it would only make sense for people following that lifecycle. For example, Scrum terminology makes sense if you’re following the Scrum-based Agile/Basic lifecycle but not the Lean Continuous Delivery lifecycle.
3. Terms should already be in use elsewhere. We are not in the business of creating new terms when existing ones are perfectly fine.

The Problem with Scrum Terminology

Many people ask us why we don’t simply use Scrum terminology. We originally wanted to, because that would be the easy thing to do, but we quickly realized that Scrum terminology just doesn’t get the job done for three reasons:

1. It doesn’t apply in all situations. For example, the term “sprint retrospective” doesn’t really make sense when you’re following a lean lifecycle that doesn’t have the concept of sprints/iterations. Furthermore, it breaks principle #3 above in that the Scrum folks tacked “sprint “onto the front of the existing term “retrospective” to brand it with Scrum marketing.
2. It was motivated by marketing reasons. The Scrum originators purposely chose unusual terms such as sprint, Scrum Master (later concatenated to ScrumMaster), and Scrum meeting to signal to people that Scrum was different. Well, in DA we’re purposely choosing pragmatic terminology to signal to people that it’s time to up our game as software professionals.
3. It reflects 1990s thinking. There’s nothing wrong with that per se, other than the fact that we have learned a lot the following decades that we can apply.

Mapping Scrum to Disciplined Agile Terms

The following table maps common Scrum terms to the terms that we prefer in DA. As you can see, the mapping is very straightforward.

Parting Thoughts

There is no standard terminology in the agile world, nor will their ever be. Your team, as part of owning your process, will need to decide which terms they prefer to use. We’ve seen many DA teams choose to use Scrum terminology (e.g. sprint instead of iteration) because they originally started with Scrum and that’s what they’re familiar with. That’s their decision and as always our advice is for a team to do what they believe to be right for the situation that they find themselves in.

One of the things that a delivery team needs to do, often in collaboration with product management, is choose the release cadence of their product. This is an important aspect of, you guessed it, release planning.  Your release cadence defines how often you release your solution both internally and externally into production (or the marketplace). The issue is how to determine how often the product should be released into production. In this blog we explore:

1. Where are you deploying to?
2. What affects release cadence?
3. What release cadence choices do you have?
4. What do we recommend?

Where Are You Deploying?

There are several target environments that you might choose to deploy to. These environments include:

1. Demo environment(s). Many teams maintain a demo environment for their solution so that their stakeholders can see what has been developed to date at their leisure. Demo environments support transparency with your stakeholders, reduce the number of “one-off” requests by stakeholders for demos (because they can simply see the solution for themselves), and of course they provide a stable environment in which your teams can run demos.
2. Testing environment(s). Many teams have their own testing environments, or they work with independent test teams with their own testing environments, or both. You should strive to test as often and early as possible, an implication being that want to deploy into your test environment(s) as often as you possibly can.
3. Production/marketplace. Some teams will release their solutions into their production environments (or to someone else’s cloud) where end users will use the systems. In the case of commercial software companies they will release their solutions into the marketplace where they are then sold to customers. Throughout this blog whenever we use the term production we also mean the marketplace.

For the sake of terminology, deploying into demo or testing environments are often referred to as internal releases and into production as an external release.

What Affects Release Cadence?

There are several factors that affect the choice of release cadence:

1. Stakeholder needs. How often do your stakeholders, and in particular your end users, want your solution to be released? This can be a difficult issue because very often your stakeholders might not be able to perceive what is appropriate for them. We’ve seen stakeholders ask for quarterly releases, be delighted when then get monthly releases, and then start asking for weekly releases once they realize the potential of modern agile strategies.
2. Stakeholder capability to accept change. We like to think that more often is better, and in the vast majority of situations it is. As difficult to believe as this may seem, at the far extreme we’ve also seen some systems where the natural release cadence is once every three or four years because that’s the rate at which stakeholders are able to accept change. In this case the product was a transaction processing (TP) system infrastructure product, but we’ve heard similar stories about major database management systems (DBMSs) products too. Granted, a release cadence this long is very rare but it does happen in a small number of situations. Far more common is the mistaken belief by IT professionals that their stakeholders are unwilling or unable to accept shorter release cycles. We’ve seen numerous organizations where the IT people tell us that their stakeholders can’t handle anything more regular than a quarterly or bi-annual release, yet these same stakeholders regularly use commercial software that is updated several times a week.
3. Your organizational culture. Some organizations, particularly those with an existing traditional release management team, often have release cultures that lean towards larger and less frequent releases. These organizations often have significant investments in legacy systems and insufficient investments in automated regression tests/checks. As a result releasing solutions into production tend to be seen as a risky endeavour. At the other extreme we’ve seen companies with more of a continuous delivery mindset that have a “release as swiftly and often as you can” culture. These organizations have typically invested heavily in code quality, automated regression testing, and automated deployment thus making deployment a simple and virtually risk-free effort.
4. The team’s ability to deliver. Of course a primary determinant of your release cadence will be how often you’re able to actually produce a potentially consumable solution. This is affected by the skills of your team members, your ability to collaborate, your ability to vertically slice functionality into small features, and your delivery infrastructure.
5. Your delivery infrastructure. How easy it is to release a potentially consumable solution into production is determined in part by your technical environment. This includes the extent of your automated regression tests, your automated deployment scripts, and your capability to monitor production. In general, the greater the level of automation the more often you can release.
6. Your solution architecture. Is your solution architected to be released incrementally? Is it possible to enable/disable functionality at a granular level (perhaps via feature toggles or a similar technique)?
7. The cost/risk to release.   Cost and risk tend to go hand-in-hand when it comes to releasing solutions into production. This is because the more manual your release/deployment processes the more expensive they become and the more likely there are to be problems somewhere in the process. Conversely, the more you automate the overall deployment effort the cheaper it is to deploy and the less risky it becomes as you’re more likely to run into, and then automate a solution to, deployment problems. The less expensive and less risky it is to release your solution the more viable it becomes to release more often.
8. Release cadence of other teams. Like it or not your team is likely dependent on the work of other teams. For example you may need web services being built by another team, and you can’t fully release your solution until those web services are available in production.  We’ve written detailed articles about how to manage dependencies between agile/lean and traditional teams, dependencies between agile teams, and dependencies between agile and lean teams.

Release Cadence Choices

Table 1 lists many common release cadences, from more than annual to several times a day. It also lists the potential tradeoffs of each approach and indicates when you may want to adopt each one.

Table 1. Comparing external release cadence options.

Our Recommendations

When it comes to releasing your solution, we have several recommendations for you to consider:

1. Automate, automate, automate. The more you have automated, the lower the cost of deployment and the lower the risk. This enables you to release more often with confidence.
2. Release internally very often. This is your opportunity to get good at releasing your solution, at squeezing out the cost and the risk.
3. Release externally as often as possible. The faster and more often you can release into production the more competitive your organization will be.
4. Always look for ways to release more often. Impressed with your ability to release once a month? Aim for bi-weekly. You’ve now releasing bi-weekly? What’s stopping you from releasing weekly? Weekly releases? Meh! Release daily! Your team is releasing daily grandpa? How about automatically releasing many times a day every time you have a working build?

Further Reading

• The Release Management Process blade describes the issues surrounding releases in a multi-team enterprise environment.

We often hear the refrain that Disciplined Agile (DA) is too complicated, that it needs to be simpler. Believe me, we’re all for simplicity and we do everything that we can to make DA as simple as possible. However, to be fair, we often find that the people complaining about this are often coming at the situation from a different point of view than we are.

With this blog we work through how why the complications encompassed by DA reflects the reality that we face in modern enterprises. To do this let’s explore why DA may appear to be complicated at first. This blog explores the following issues:

1. There is a natural complications in what we do
2. Enterprise solution delivery is inherently complicated
3. Enterprise IT is inherently complicated
4. Choice is good, but having choices adds complications
5. People underestimate the inherent complications of the familiar
6. DA purposefully includes some questionable choices
7. Most people prefer to focus on a small part of the overall process
8. What would you take away?
9. What would you like to add?

There is a Natural Complications in What We Do

Software development by itself is complicated: we need to work closely with stakeholders to understand their needs, we need to architect and design solutions, we need to build or configure these solutions, we need to test them, we need to deploy them, we need to organize this effort somehow, and many other things. There’s clearly a bit of natural complexity in software development. Furthermore, the overall IT process surrounding software development adds even more complications to operate, manage, and govern our overall IT ecosystem.

This is why we get paid what we get paid – if it was easy, organizations could find a multitude of lower-skilled, and lower-paid, people to do this work.

Enterprise Solution Delivery is Inherently Complicated

The Disciplined Agile Delivery (DAD) portion of the DA toolkit addresses solution delivery in enterprise-class organizations. Sometimes people who have only worked in startups or small companies don’t appreciate the realities of enterprise-class development. A small company that has been in business for a few years is very different than a large organization that has several decades of legacy systems, legacy processes, and legacy thinking in place. Adopting agile and lean ways of working when you effectively have a clean slate to start from is a very different proposition than working in an organization where you need to respect and evolve the existing legacy ecosystem.

But don’t other, simpler methods such as Scrum work in these settings? Yes, they potentially do, but they address a much smaller scope than does DAD. Scrum is less complicated because it only looks at a very small portion, likely not even 5%, of the software delivery process. The focus of Scrum is on a few collaboration patterns for organizing small software construction teams and for managing changing requirements. It purposefully does not address the full project delivery lifecycle from beginning to end. Nor does Scrum address technical practices for architecture, design, testing, programming, modeling, documentation, deployment, and many other aspects of solution delivery. DAD, on the other hand, purposely addresses all of these aspects of software delivery from beginning to end in a streamlined manner. And Scrum only describes a single way of working, what DAD calls the Agile/Basic lifecycle, whereas DAD includes six lifecycles (Agile, Lean, Continuous Delivery: Agile. Continuous Delivery: Lean, Program, and Exploratory/Lean Start Up) to choose from to address a great range of needs. So yes, as a toolkit DAD is much more complex than Scrum because DAD answers the multitude of questions that Scrum leaves up to you.

Interestingly, from the point of view of enterprise agile transformation, DAD is a less complicated solution to adopt because it’s already done the “heavy lifting” for you on the process side of things.  We wrote a blog a few years ago about how it’s easier to right-size your process by starting in the middle with something like DAD than it is to start at the “simple bottom” with Scrum.

Enterprise IT is Inherently Complicated

Disciplined Agile IT (DAIT) extends DAD (and Disciplined DevOps) to look at all of IT. The workflow of a Disciplined Agile IT department is depicted in the following diagram, and that diagram is merely a high-level overview. Many of the challenges that developers like to complain about – dysfunctional governance, ineffective enterprise architecture, slow data management, non-existent reuse, questionable HR processes, and many more – DA chooses to explicitly address. Because DA deals with the much wider range of enterprise IT issues than just software development it encompasses more complexity than many developers may be comfortable with, or even aware of.

DA addresses this bigger picture because our philosophy is to provide coherent, pragmatic advice to help organizations address the challenges that they face in improving their processes. In fact, the DA toolkit appears to be the only source of information that addresses enterprise agile IT in a coherent and comprehensive manner.   Without a holistic vision such as this an organization will end up development a piece-meal strategy where the individual parts may be effective on their own but do not integrate well as a whole. Not looking at the entire IT picture at once is the primary cause of organizational dysfunctional in your department today, with the DA toolkit you can avoid this problem as you evolve into an agile/lean way of working.  BTW, you can download a copy of the workflow diagram shown above from the Disciplined Agile posters page.

Choice is Good, But Having Choices Adds Complications

Disciplined Agile looks at a range of situations, giving you choices. Other methods and frameworks, such as LeSS and SAFe, prescribe a single way of working.   Certainly a small development team will work differently than a medium-sized development team which works differently than a large team. A team that is co-located will work differently than a team spread across a single building, than a team spread across several buildings in the same city, than a team spread across the world. A team working in a regulatory environment will work in a more sophisticated way than a team in a non-regulatory environment. A team where everyone works for your company will work differently than a team where some of the work is being outsourced. A team facing a simple domain problem will work differently than a team facing a complex problem. And so on. Although many people just want to be told what to do, they just want a single way of working given to them, that desire for prescription is a recipe for disaster in modern IT organizations. You have a range of teams facing a range of situations: One process size does not fit all.

Having choices is definitely complicated. This is similar to “needing a new outfit”, going to the store, only to be overwhelmed by the myriad of choices presented to you.

One way to deal with the this is to divide and conquer. Recognize that deciding on an outfit that is right for you is actually a collection of simpler choices. For example, your need to choose the right shirt, the right pants, the right shoes, and so on. Of course you shouldn’t make these choices in isolation because your outfit needs to work together as a whole.  You either need to coordinate your decisions and effectively make them in parallel or if you prefer to make decisions one at a time then be prepared for your shirt choice to limit your follow-on clothing choices.

Let’s bring the conversation back to software process. In Disciplined Agile Delivery (DAD) we called out a collection of process goals that a team should fulfill in some way to be successful at agile solution delivery. These goals included exploring the initial scope, putting the team together, developing a consumable solution, coordinating activities, and deploying the solution into production to name a few. Each process goal is overviewed by a process goal diagram, the one for Explore Initial Scope follows below. Each goal is organized into a collection of process decision points, each decision point has options, and each option has advantages and disadvantages. Where one agile method might advise you to write a collection of epics and user stories to identify the initial scope, in DAD we say that you need to explore how people will use your solution (and writing epics and user stories are two of many options available to you), you may need to explore the user interface, you should consider to what extent (if any) the initial requirements should be documented, how you will go about eliciting requirements, and other important decisions. Ideally, if you want to be effective, then you want each team to own their own process and make the choices that are best for them given the situation that you face.

Back to the clothing analogy. Just like Scrum and SAFe prescribe one way of working, you could similarly take a prescriptive approach to shopping for new clothes. This is the equivalent of saying that you will be successful if you wear a red polo shirt, blue pants, blue socks, and black shoes. That may be great advice if you are working as a food server at your local restaurant chain but horrible advice if you need an outfit that is appropriate for a wedding, or for working in a formal office, or for going to the beach. When buying the right clothing you need first need to understand the context and then hopefully have several choices available to you. You want to be able to choose the right shirt for you, not the one prescribed to you. You want to choose a nice pair of pants or a skirt that complements that shirt, and so on. Yes, it’s much easier just to buy exactly the clothes that you’re told to, but it doesn’t give you as satisfying a result in most situations.

In Disciplined Agile we extended this choice-driven approach to IT-level activities such as Portfolio Management, Enterprise Architecture, Reuse Engineering, Data Management, and others. The following diagram is the process goal diagram for Enterprise Architecture, depicting the process decision points that your EA team should consider addressing and a range of options for each choice.

People Underestimate the Inherent Complications of the Familiar

When most people are asked what the gentleman in the picture is wearing, they’ll say a blue dress shirt and slacks. However, he’s also wearing shoes, socks, a belt, and underwear (we hope).   It is quite common for people to abstract away, to ignore, details that are familiar to them – the “shirt and slacks” answer abstracted away the details of the entire outfit. However, for someone to whom western clothing is new, this “simple outfit” appears quite complex to them until someone helps them to understand it.

The exact same thing happens in the software world. When asked what we do it’s easy to casually say that we work with stakeholders to understand what they want, we write some software, and then we deploy it. Of course the reality of what actually goes on is a lot  more complex than that.

We invite you to spend a few minutes and jot down all of the activities that your team does to actually build and deploy software in practice. Pretty long list, isn’t it? And for every strategy on that list, you’ve selected one of many options available to you.

DA Purposefully Includes Some Questionable Choices

We all make choices in our diet. Some of those choices are good and some of those choices are rather bad for us. We know that we’re making poor choices yet we still choose to do so, perhaps because that’s the best we can do at the time or simply because that’s what we feel like eating right now.

It’s exactly the same when it comes to process – we have a range of choices and some of them are better for us than others. For example, consider the choices we have when it comes to the level of detail that we capture when exploring the initial scope. We could choose to: write a very detailed requirements specification; write a light specification (perhaps using index cards, whiteboard sketches, and a bit of overview documentation); write a simple bulleted list of goals; or not capture any requirements information at all. Many agile purists will say that writing a detailed requirement specification isn’t agile, and in the vast majority of situations it isn’t, although we have seen life-critical regulatory situations where it is appropriate.   But, what if you’re in a situation where detailed requirements specifications aren’t appropriate yet someone is demanding that you create one? Agile purists will likely get into religious arguments with this person: the traditionalist will argue that their way is best and the purist will argue that there way is best, but in the end the person with the most political clout will win. With a DAD-based we recognize that there is a range of options available to you, including writing detailed specifications. But we take it one step further and are clear about the advantages and disadvantages of each approach and when the approach is appropriate.   Now you can have more intelligent conversations and pragmatic about your process choices.   When you get into situations such as this you can go beyond the “but that isn’t agile” refrain to have coherent reasons that as to why one strategy is better for another strategy given the situation that you face. Choice is good.

Most People Focus on a Small Part of the Overall Process

Many people choose to focus on their part of the job, which is fair enough. But what about all of the other activities that your team performs from beginning to end? You may be a developer focused on software construction, but what about all of the initial inception work that occurred, such as gathering initial requirements, putting the team together, and getting funding for construction that occurred before you began your work? What about the activities that occur within other teams that interact with, often to support, your team? For example, there may be some enterprise architects responsible for evolving and supporting a technical roadmap for your organization, there may be an infrastructure team responsible for operating your technical environment, and there may be people responsible for people management/HR activities (such as seeing that you get paid). Disciplined Agile looks at the holistic picture, not just at your piece of the overall puzzle. Just because you’re not involved with data management, or portfolio management, or operations doesn’t mean that those activities, and the teams that perform them, aren’t important to the overall success of your organization.

What Would You Take Away?

Now that you understand the overall scope of DA, what would you remove? We’re asking this question from the point of view of:

• The complexity faced by modern enterprises
• That you need choices if you’re to actually own your process
• That we’re considering the overall picture, not just your part of it

If you think Disciplined Agile (DA) contains something that it doesn’t need, please add a comment at the end of this blog.

What Would You Like to Add?

Ha! Every single time we’ve had this conversation with someone, and they were willing to consider the issues above, they’ve told us that we needed to include one or more techniques that they’ve found effective in practice. EVERY. SINGLE. TIME.  In short, the conversation goes from DA is too complex to DA is not complex enough.  Argh!

So, once again, please add a comment at the end of the blog for anything you think we’re missing.

A Few Parting Thoughts

Software development, IT in general, and in fact your organization is a complicated game with many moving parts that are evolving all of the time. It’s natural for us, as human beings, to want simple answers. But sometimes the simplest answer can still be quite complicated. Simplistic answers may be more attractive, and in the short term or in very narrow situations they may in fact be a good solution for you, but more often than not they’ll make your situation even worse.   In Disciplined Agile (DA) we actively strive to drive the complications out of the software process but at the same time we have the courage to look at the bigger picture and explicitly address the range of issues that modern enterprises face in practice. With DA we’re working together to make the process as simple and streamlined as possible.