Disciplined Agile

A common question we get regarding Disciplined Agile (DA) is “What makes DA more disciplined than other approaches to agile?”  It’s a fair question, particularly from someone who is new to DA.  This blog posting explores this question, explicitly summarizing the critical strategies that exhibit the greater levels of discipline in DA as compared with what we see in many agile teams today.

It is clear that many common agile practices require discipline.  For example, agile teams it takes discipline to hold concise, streamlined coordination/Scrum meetings; to consistently deliver business value every iteration; to test continuously throughout the lifecycle; to improve your process “in flight”; to work closely with stakeholders and many more things.  Discipline is very important to the success of agile teams, see The Discipline of Agile for a detailed discussion, and DA takes it to the next level in the following ways:

1. Reducing the feedback cycle. Techniques that shorten the time between doing something and getting feedback about it are generally lower risk and result in lower cost to address any changes than techniques with longer feedback cycles.  Many of these techniques require agile team members to have new skills and to take a more disciplined approach to their work than they may have in less-than-agile situations.  There are several common ways to shorten the feedback cycle that are common to agile software development that are adopted by DA.  These techniques, listed in order of immediacy, include non-solo development (e.g. pair programming), active stakeholder participation, continuous integration (CI), continuous deployment (CD), short iterations/sprints, and short release cycles.
2. Continual learning. Continual learning is an important aspect of agile software development in general, not just DA.  However, DA explicitly addresses the need for three levels of learning: individual, team, and organizational/enterprise.  It also addresses the need for three categories of learning: domain, technical, and process.  Continual learning strategies include active stakeholder participation, coaching, mentoring, individual learning, non-solo development, experimenting via minimum viable products (MVPs), spikes, retrospectives/reflections, sharing lessons learned between teams, and stakeholder demonstrations.
3. Incremental delivery of consumable solutions.  Being able to deliver potentially shippable software increments at the end of each iteration is a good start that clearly requires discipline.  DA goes one step further and advises you to explicitly produce a potentially consumable solution every iteration, something that requires even greater discipline.  Every construction iteration your team requires the discipline to create working software that is “done”, to write deliverable documentation such as operations manuals and user documentation, to address consumability (usability), to consider organizational change issues pertaining to your solution, and operations and support issues (an aspect of DevOps).
4. Being process goal-driven.  The DA tool kit promotes a process goal-driven approach. For each goal we describe the issues pertaining to that the goal. For example, with initial planning you need to consider issues such as the amount of initial detail you intend to capture, the amount of ongoing detail throughout the project, the length of iterations, how you will communicate the schedule (if at all), and how you will produce an initial cost estimate (if at all). Each issue can be addressed by several strategies, each of which has trade-offs. Our experience is that this goals-driven, suggestive approach provides just enough guidance for solution delivery teams while being sufficiently flexible so that teams can tailor the process to address the context of the situation in which they find themselves in. The challenge is that it requires significant discipline by agile teams to consider the issues around each goal and then choose the strategy which that is most appropriate for them.
5. Enterprise awareness.  Whether you like it or not, as you adopt agile you will constrained by the organizational ecosystem, and you need to act accordingly.  It takes discipline to be enterprise aware and to work with enterprise folks who may not be completely agile yet, and have the patience to help them.  It takes discipline to work with your operations and support staff in a “DevOps” manner throughout the lifecycle, particularly when they may not be motivated to do so.  Despite the fact that governing bodies such as project management offices (PMOs), architecture and database authorities, and operations may indeed be a source of impediments to your DA adoption, or they may be your greatest allies, these authorities serve important functions in any large enterprise.  Therefore a disciplined approach to proactively working with them and being a positive change agent to make collaboration with them more effective is required.
6. Adopting a full delivery lifecycle.  Despite some agilists reluctance to admit that projects go through phases, DA explicitly recognizes that they do.  Building serious solutions requires a lot more than just doing the cool construction stuff.  It takes discipline to ignore this rhetoric and frame your project within the scope of a full delivery lifecycle.  The two project-based life cycles explicitly depict pre-delivery activities, a three-phase delivery lifecycle, and post-delivery activities (operations and support).
7. Streamlining inception activities.  The project life cycles include an Inception phase, known as Sprint 0 in Scrum, where you perform fundamental project initiation work.  Unfortunately in our experience we have seen many organizations treat this phase as an opportunity to do massive amounts of upfront documentation in the form of project plans, charters, and requirements specifications.  Some people have referred to the practice of doing too much transitory documentation up front on an agile project as Water-Scrum-Fall.  We cannot stress enough that this is NOT the intent of the Inception phase.  While we provide many alternatives for documenting your vision in Inception, from very heavy to very light, you should take a minimalist approach to the Inception phase and strive to reach the stakeholder consensus milestone as quickly as possible.  If you are spending more than a few weeks on this phase, you may be regressing to a Water-Scrum-Fall approach.  It takes discipline to be aware of this trap and to streamline your approach as much as possible.
8. Streamlining transition activities.  In most mid-to-large sized organizations the deployment of solutions is carefully controlled, particularly when the solutions share architectures and have project interdependencies.  For these reasons release cycles to your stakeholders are less frequent that you would like because of existing complexities within the environment.  However, the ability to frequently deploy value to your stakeholders is a competitive advantage; therefore you should reduce the release cycle as much as possible.  This requires a great degree of discipline in areas such as pre-production integration and deployment testing; regular coordination between project teams and with operations and support staff; Change management around both technology and requirements; and adoption of continuous deployment practices to such a degree that very frequent deployments are the norm and the Transition “phase” becomes an automated transition activity.
9.  Adopting agile governance. It is easier to avoid your traditional governance and tell management that “agile is different” than it is to work with your governors to adapt your governance to properly guide the delivery of your agile teams.  Every organization has a necessary degree of governance and there are ways to make it especially effective on agile initiatives.  It takes discipline to work with your governors to help them understand how disciplined agile teams operate and then discipline to accept and conform to the resulting governance process.
10. Moving to lean. For all of the process goals we describe a range of options to address the issues pertaining to that goal.  These options ranged from traditional/heavier approaches that we generally advised against except in very specific situations to agile strategies to very lean strategies.  Generally, the leaner the strategy the greater the discipline it requires.

Adopting a disciplined approach to agile delivery requires the courage to rethink some of the agile rhetoric and make compromises where necessary for the benefit of the “whole enterprise” and not just the whole team.  In our experience most agile projects make certain compromises that are not classically agile in order to get the job done.  Rather than hiding this and fearing reprisals from those who would accuse you of regressing to a traditional approach, it is better to have the courage to take a pragmatic approach to using agile in your situation.

Effective application of DA certainly requires discipline and skill, but in our experience the key determinant of success is the ability and willingness of the team to work well together and with  stakeholders, both within and external to the team.  

A common question that we get is whether it’s possible for a team to take an agile approach in a regulatory environment.  The answer of course is a resounding yes, although your approach will need to be tailored to reflect the constraints of the applicable regulation(s).

Let’s explore issues pertaining to compliance:

1. The regulations vary.  Not all regulations are created equal.  For example, financial regulations such as Sarbanes Oxley (SoX) are typically less stringent than life-critical things such as the various Federal Drug Administration (FDA) regulations.  So, one regulatory compliancy strategy does not fit all and your team will instead need to tailor their agile strategy to reflect the applicable regulations that you face.
2. Agile teams are working in a regulatory compliance scenarios.  The quick answer is yes.  As you can see in the chart above, the 2016 Agility at Scale study found that two-thirds of agile teams face either regulatory, organizational, or both forms of compliance.
3. Organizations are succeeding at applying agile within a regulatory regime.  The 2012 Agility at Scale study found that some respondents indicated that their organizations had successfully applied agile strategies with regulatory situations. As you can see in the chart below they are applying agile in all types of regulatory environments, including but not limited to life-critical and financial.  If other organizations are succeeding at doing so perhaps yours can as well.
4. Organizations are failing at this too.  The 2012 Agility at Scale study also asked if organizations had agile project teams that failed within regulatory situations and respondents indicated that they had.  If other organizations are struggling with agile and regulatory compliance then yours might too, so please consider the advice provided below.
5. The regulations rarely tell you how to work.  Regulations typically provide criteria that your process needs to meet.  For example they may call out the need to have independent testing, but they won’t say that you need to have an onerous testing phase nor that all testing needs to be done this way.  There you could adopt parallel independent testing in addition to your whole team testing efforts to conform to this requirement.  The implication is that you can tailor your solution delivery process to be as agile as you can while still being compliant – you don’t need to take a waterfall/V-model style approach.
6. Sometimes compliancy is self imposed.   Some compliancy requirements are not legislated, such as FDA and SoX, but are instead willingly adopted by your organization.  Examples of this include compliancy regimes such as ISO-900X and CMMI, strategies which may have been adopted for marketing reasons (typically by IT service providers) or perhaps process improvement reasons.  As you can see in the chart organizations are both succeeding and failing at applying agile in these situations.
7. You need to read the regulations.  Our experience is that many organizations will let their more bureaucratic-leaning staff members interpret how to conform to regulations.  Not surprisingly their strategy often involves a lot more paperwork, activities, and checkpoints than is actually needed.  When pragmatic people are asked to interpret regulations you often end up with a more pragramatic response.  So, if you’re in a regulatory environment we’ve found that it behooves you to take the time to read the regulations so that you can streamline how your agile team addresses them.  Fair warning: Most regulations are incredibly dry reading.

Disciplined Agile Delivery (DAD) addresses regulatory compliance issues via several key strategies:

1. Adopt a hybrid process.  DAD is a hybrid tool kit that adopts strategies from a range of sources including Scrum, XP, Agile Modeling, Kanban, Unified Process, and many more.  Regulations typically cover a wide range of issues and as a result you need to adopt supporting practices from numerous sources.  This may include management practices from Scrum, agile development practices from XP, agile documentation practices from Agile Modeling, data quality practices from Agile Data, and so on.  The DA toolkit has already done the heavy lifting for you by showing how these practices fit together, unlike methods such as Scrum which leave this work up to you.
2. Adopt a full delivery lifecycle.  Most regulations address the full delivery lifecycle, not just construction.  DAD supports a full delivery lifecyle, in fact it supports several such lifecycles (a Scrum-based lifecycle, a lean lifecycle, a continuous delivery lifecycle, and so on) to reflect the differing contexts faced by teams in typical enterprise environments.
3. Focus on solutions, not just software.  Disciplined agile teams produce consumable solutions, not just “shippable software”.  DAD recognizes that delivery teams are working on solutions that have a software component, that run on hardware, that are supported by documentation, and that the team may even change the business process around the usage of a system and even the organization structure of the people using it.
4. Take a goal-driven approach. Recognizing that solution delivery teams find themselves in unique situations, DAD doesn’t prescribe how they should work.  Instead, it focuses on providing advice for how teams can tailor their strategy to reflect that context of the situation that they find themselves in.  DAD does this by promoting a process goal driven approach.  This strategy guides teams through the process decisions that they’re making, some of which will be driven by regulatory compliance.  The DA tool kit has already done a lot of the heavy lifting regarding how to tailor your agile process to meeting scaling concerns such as regulatory compliance, large teams, geographically distributed teams, and other issues.  
5. Adopt an explicit governance strategy.  DAD has agile governance strategies built right in, including explicit light-weight milestones, metrics, named phases, and many other aspects of governance expected by many regulations.  Once again, DAD has done a lot of the heavy lifting for you.
6. Be enterprise aware.  DAD promotes the concept of enterprise awareness, the recognition that agile teams do not work in a vacuum.  This includes strategies for engaging with enterprise architects, how to deal with enhancement requests and defect reports coming in from operations, and how to work with other enterprise professionals.  These can be key issues to understand when tailoring agile to be compliant within an existing organizational ecosystem – your entire process needs to comply to the regulations, not just the development portion of it.

In short, yes it is possible to successfully follow a disciplined agile strategy given the constraints of regulatory compliance.  

A few people have commented that Disciplined Agile Delivery (DAD) promotes a wide range of practices, which they like because it makes their options explicit but which they also potentially dislike because there’s so many practices to choose from.  This then leads to the question of why do we need so many practices?  First, there are a lot of practices out there to begin with and our philosophy is to help people know that they have options and help them to select the right ones.  Second, our experience is that for a practice to be easily consumable it should be:

1. Small.  Small practices are generally more straightforward than larger practices.  As a result they’re easier to understand and to adopt.
2. Cohesive.  A good practice addresses one issue and one issue only.
3. Loosely coupled.  Good practices, like good software modules, have minimal dependencies on other practices.

Practices that are small, cohesive, and loosely coupled are easier to configure into more interesting solutions.  For example, the practice of test-first programming (TFP) is combined with refactoring to form test-driven development (TDD).  The practice of (writing) executable specifications can be combined with TDD, or TFP for that matter, to give you behavior driven development (BDD) or acceptance test-driven development (ATDD).  The combination of iteration modeling, model storming, look-ahead modeling, and BDD can give you a strategy for addressing emergent requirements and design during an iteration.

Of these three aspects, we’ve found that coupling has the greatest impact on your ability to tailor your approach to meet the unique situation you find yourself in.  Just like highly coupled software is difficult to maintain and enhance, processes built from highly-coupled practices are too.  For example, consider the way that Scrum describes product backlogs.  A product backlog is one of several strategies that agile teams may use to manage their work.  In the case of Scrum, the strategy is to prioritize requirements by business value and then focus on implementing the highest priority work at all times.   Unfortunately Scrum has coupled many important practices to the product backlog concept.  For example, initial requirements modeling is often referred to as populating the backlog.  Prioritization of new requirements and exploring upcoming requirements is referred to as grooming the backlog.   There are several potential problems to consider:

1. The term “populating the backlog” masks the fact that not only are you writing initial functional requirements (for example user stories or features) as part of your initial requirements modeling efforts you’re also sketching things (processes, screens, data structures, …), identifying non-functional requirements, holding modeling sessions, and many other things.
2. It makes it harder for people new to agile to understand how it works.  Think of it like this, what’s a more descriptive term, “populating the backlog” or “initial requirements modeling”?
3. It makes it harder to combine practices.  If you wanted to swap out the product backlog for something a bit more sophisticated, such as a work item list, or something leaner like a work item pool, what do you do with the practices coupled to product backlog?  Do you  rework “backlog grooming” to be “work item list grooming”?  Do you rework “populating the backlog” to be “populating the work item pool”?  Even if these things are easy to do, seems like needless effort to me.

In conclusion, we have found that adopting small, cohesive, and loosely coupled practices enables you to adopt and tailor a process strategy that better reflects the context of the situation that you face.    Not only is high cohesion and loose coupling great strategies for software, their great strategies for software process too.

Recently on the Disciplined Agile Delivery LinkedIn discussion group we talked about how people in an existing traditional role fit into a disciplined agile team.  The challenge is that in organizations new to agile will have people who take on traditional roles such as business analyst, programmer, solution architecture, tester, project manager and so on.  Yet DAD has roles such as team member, team lead, architecture owner, and so on that are different from traditional roles.  Clearly this is a “DAD adoption” challenge that we need to overcome.

At first you will build cross functional agile teams with the people you have available to you.  Agile teams are whole teams, which means that the team has sufficient skills to get the job done.  The implication is that you’ll initially build a team from the analysts, programmers, testers, … that are available and are willing to join the team.  As they say, you go to war with the army that you have.  At first an analyst is likely to focus on analysis activities, a programmer on programming activities, and so on because that’s what they know how to do.  Because they are working closely with one another in a iterative, incremental, and collaborative manner they quickly start to pick up skills from one another.  Over time these people who were originally specialized in one aspect of solution delivery become T-skilled generalizing specialists with a wider range of abilities.  This enables them to collaborate more easily and be more effective as IT professionals.

An interesting side effect of this is that as your team becomes more experienced in working in an agile way, and as team members gain a wider range of skills, the conversation shifts from “I’m a tester, so I’m responsible for testing” to “as a team how are we going to approach testing X?”  In other words, you start to focus on performing valuable activities instead of the roles responsible for doing so.  This is an important shift in mindset on your agile transformation journey.

When a disciplined agile project or product team starts one of the process goals which they will likely need to address is Explore Initial Scope.  This is sometimes referred to as initially populating the backlog in the Scrum community, but as you’ll soon see there is far more to it than just doing that.  This is an important goal for several reasons.  First, your team needs to have at least a high level understanding of what they’re trying to achieve, they just don’t start coding.  Second, in the vast majority of organizations IT delivery teams are asked fundamental questions such as what are you trying to achieve, how long will it take, and how much will it cost.  Having an understanding of the scope of your effort is important input into answering those sorts of questions.

The process goal diagram for Explore Scope is shown below.  The rounded rectangle indicates the goal, the squared rectangles indicate issues or process factors that you may need to consider, and the lists in the right hand column represent potential strategies or practices that you may choose to adopt to address those issues.  The lists with an arrow to the left are ordered, indicating that in general the options at the top of the list are more preferable from an agile point of view than the options towards the bottom.  The highlighted options (bolded and italicized) indicate default starting points for teams looking for a good place to start but who don’t want to invest a lot of time in process tailoring right now.  Each of these practices/strategies has advantages and disadvantages, and none are perfect in all situations, which is why it is important to understand the options you have available to you.

Let’s consider each process intent/decision point:

• Choose the level of detail.  How much effort should you put into capturing the requirements, if any at all.  A small co-located team may find that capturing user stories on index cards to be sufficient, whereas a team that is geographically distributed across several locations will find that it needs to capture point-form notes about each story using an electronic tool, and a team in a life-critical regulatory environment may need to capture even more detail to the point of doing big requirements up front (BRUF).
• Explore usage. Although much ado has been made of user stories, and they can be applied quite effectively in a range of situations, the fact is that they’re only one of several options for your team to explore usage of the solution (scenarios, personas, and use cases being other options).
• Explore the domain. Some teams will choose to do some domain modeling via a data model or class diagram, as well as address other views as appropriate.
• Explore the process. Many teams will discover that they need to explore the overall workflow, or business process, supported by their solution so as to help them better understand their usage requirements.
• Explore user interface (UI) needs. Many agile teams will also choose to create user interface prototypes, either low-fidelity UI prototypes using paper or even high-fidelity UI prototypes using a prototyping tool or code, particularly when they face a complex domain.
• Explore general requirements.  There are several types of functional requirements modeling techniques that can be valuable that don’t fit well into the previous categories.
• Explore non-functional requirements.  How will non-functional requirements pertaining to availablity, security, performance, and many other issues be addressed?  Teams in straightforward situations may find that capturing them as technical stories may be sufficient.  Teams facing technical complexity, and sometimes even domain complexity, soon discover that they need a more sophisticated strategy.
• Apply modeling strategy(ies).  How will your team go about working with stakeholders to elicit/discover their perceived needs?  Will they hold informal modeling sessions in an agile modeling space?  Will they hold formal modeling sessions, perhaps following a Joint Application Design (JAD) strategy?  Will they interview people one-on-one?  Combinations thereof?
• Choose a work item management strategy.  Early in the project you will want to determine how you intend to address changing stakeholder needs throughout the project as this will affect how you address the other process issues in this list.  For example, do you intend to adopt Scrum’s value-driven product backlog strategy, DAD’s risk-value driven work item list, a lean work item pool strategy (as followed by DAD’s lean lifecycle), or even a formal approach?  A team in a strict regulatory environment may be required to have a more formal approach to change management than a team without this restriction.

I wanted to share two important observations about this goal.  First, this goal, along with Identify Architecture Strategy, Coordinate Activities, and Accelerate Value Delivery seem to take the brunt of your process tailoring efforts when working at scale.  It really does seem to be one of those Pareto situations where 20% addresses 80% of the work, more on this in a future blog posting.  As you saw in the discussion of the process issues, the process tailoring decisions that you make regarding this goal will vary greatly based on the various scaling factors.  Second, as with all process goal diagrams, the one above doesn’t provide an exhaustive list of options although it does provide a pretty good start.

I’m a firm believer that a team should choose their own way of working, including their team structure, their work environment, and their process, to reflect the situation that they find themselves in.  When it comes to process tailoring, process goal diagrams not only help teams to identify the issues they need to consider they also summarize potential options available to them.  Agile teams with a minimal bit of process guidance such as this are in a much better situation to tailor their approach that teams that are trying to figure it out on their own.  The DA tool kit provides this guidance.

Related Reading

• Guided Continuous Improvement (GCI)
• Choosing Your WoW: The Situation Context Framework (SCF)