Disciplined Agile

Recently on the DAD LinkedIn discussion forum we’ve had an interesting conversation going on that strayed into the effectiveness of various communication techniques.  This conversation got me to thinking a bit about an article that I first wrote over a decade ago, Communication on Agile Software Projects, based on material in my Agile Modeling book.  This article extends some of Alistair Cockburn’s thinking on the topic, summarized in Figure 1, which in turn extends a body of work called Media Richness Theory.  The basic observation is that the richer the communication channel – such as email, videoconferencing, or face-to-face conversation – the more effective it is in practice.

Figure 1. Comparing the effectiveness of communication techniques.

In the DAD book Mark and I said that the implication of Figure 1 was that given the situation you find yourself in that you should choose to use the most effective communication strategy available to you.  We thought that this was all that needed to be said, but since then we’ve come to realize that a better diagram is needed.  We developed Figure 2 which we believe communicates this strategy a bit more clearly.

Figure 2. Choosing the best communication technique for your situation.

Figure 2 indicates that that the viability of communication options available to you varies depending on the situation that you find yourself in.  For example, people on a co-located team can communication in pretty much any manner they choose.  Ideally they would choose face-to-face communication around a whiteboard or paper but there’s nothing stopping someone from writing a detailed document and then emailing it to someone sitting beside them.  When a team is geographically distributed they have fewer communication strategies available to them, unless of course they invest in travel so that they can be face-to-face.

Figure 2 also indicates strategies for capturing, or persisting, a conversation.  For example, a few people could have a conversation around a whiteboard, sketching the design of a screen perhaps.  When they’re done they might decide to take a digital snapshot of it so that they can retain the diagram to work on it later (many organizations still don’t provide permanent whiteboard space to their development teams).  They may even choose to have someone write up notes, perhaps on a wiki or in a word processor, and embed the snapshot(s) into those notes.

To summarize, when it comes to how people communicate with one another they have choices.  Disciplined agilists prefer to choose the most appropriate communication strategy for the situation that they find themselves in.

We’ve recently been asked how risk management is addressed on agile teams. This is an easy question to answer because risk management strategies are built right into the Disciplined Agile (DA) toolkit.

Let’s start with two definitions. First, a risk is an exposure to a potentially negative outcome. Risks come about from uncertainty. Second, risk management is the identification, exploration, and mitigation of risks. Risk management should be part of both your team management efforts, even on self-organizing agile teams, as well as part of your organizations governance efforts.

Regardless of your delivery paradigm, there are several common categories of risks faced by IT delivery teams:

• Business risk. This includes significant shifts in requirements due changing business priorities, often due to realization new market opportunities, in reaction to moves made by competitors, or because you are opening completely new market opportunities. Another common risk is an overly optimistic/aggressive schedule which motivates your team to take unfortunate shortcuts. A third type of business risk is a misaligned budget: either there isn’t sufficient funding to do the job effectively (e.g. there’s no travel budget for a geographically distributed team, there’s no funding for coaching on a team new to agile, there’s no funding for training, and so on) or there is far too much funding available and the team is motivated to invest it in wasteful activities.
• Technical risk. IT teams face technical risks as the result of combining technologies in new ways, evolving technology platforms, new technologies, and lack of experience within the team in these situations.
• Operational risk. This category of risk pertains to production-oriented issues surrounding the support and operation of a solution. Will your solution work within the existing organizational ecosystem? Does your operations team have the skills and resources to run your solution? Does your support/help desk team have the experience and knowledge required?
• Process risk. As DAD clearly shows, every technique has advantages, disadvantages, and specific situations where the technique makes sense. The implication is that there are risks taken on when you apply a technique outside of its range of applicability or your comfort zone.
• Organizational risk. IT teams face organizational risks such as dysfunctional politics, competing visions, challenges pertaining to your agile transformation challenges, reorganizations, and many others.

Recognizing that IT delivery teams must deal with risks on a regular basis DAD builds in several agile risk management strategies:

• Continuous engineering practices.  This is a collection of practices with very short feedback cycles, thereby enabling you to adjust course quickly. These practices include behavior driven development (BDD), test-driven development (TDD), continuous integration (CI), continuous deployment (CD), and many others.  All of these practices require discipline and skill to adopt.  Although these practices are technical in nature, in practice they mostly reduce business risk through shortening the cycle between a stakeholder having an initial idea and the team providing a solution which reflects that idea.
• Agile architecture practices.  The DA toolkit suggests a collection of continuous architecture strategies throughout the entire lifecycle.  These include, but are not limited to, identifying an initial technical strategy, proving the architecture early in construction, deferring architecture and design decisions to the most appropriate moment, architecture spikes, just in time (JIT) model storming, and many others.  These practices reduce technical risk.
• Risk-value lifecycle.  The DA toolkit supports several lifecycles, and hence several work prioritization strategies.  One such strategy is the Unified Process (UP)’s risk-value approach where both risk and value are considered when prioritizing work items.  The end result is that work items with higher technical risk are implemented early in the lifecycle, enabling disciplined agile teams to mitigate technical risk early in the effort.  DAD teams then follow Scrum’s value-driven approach where the work is prioritized based on its business value to the organization, which is effective at addressing some forms of business risk.  The risk-value prioritization approach results in a lower overall risk profile than just the value-driven lifecycle of Scrum.  DAD also supports lean and continuous delivery versions of the lifecycle which expand upon the risk-value strategy to consider other factors such as required release dates and team health considerations.
• An explicit Inception phase.  The fundamental purpose of the Inception phase is to help your team get going in the right direction.  Executed properly, the Inception phase helps you to reduce business risk by coming to stakeholder agreement regarding the team’s strategy, technical risk by exploring a viable technical strategy, and operational risk through planning with production-staff from the very beginning.
• Risk-based, light-weight milestones.  The DAD lifecycles include explicit milestones as part of DAD’s overall governance strategy.  These milestones motivate teams to address common issues such as formulating an agreed-to vision early in the effort (business risk), proving the architecture early (technical risk), regularly considering the viability of the effort (business risk), ensuring the team has produced sufficient functionality to justify deployment (business risk), ensuring the solution is production ready (operational risk), and ensuring that the stakeholders are delighted with the result (business risk, organizational risk).
• Development intelligence.  One of the governance strategies that the DAD process framework promotes is development intelligence (DI) where a team/project dashboard is populated with metrics generated automatically by the tools the team is using.  This provides real-time insight for the team to organize itself and potentially improve its approach.  It also provides insight into your organization’s governance effort, supplying real-time data which can enable senior management to make better decisions and thereby better support your agile teams.  Development intelligence, and it’s extension IT intelligence which addresses the entire IT portfolio and processes, helps you to reduce process risk and to a lesser extent organizational risk.
• DevOps principles and practices. DAD weaves DevOps strategies through the entire framework.  This strategies are: Including operations and support staff as explicit stakeholders who work closely with the team; the lifecycle explicitly depicts production/operations; DevOps-oriented milestones (Production Ready and Delighted Stakeholders) are included as part of the governance strategy; development practices such as instrumenting solutions for operational monitoring, continuous deployment, and installation testing (to name a few); and management practices such as deployment planning. The strategies help to reduce both operational and organizational risks.
• Sophisticated go-forward decision points.  One of the business risk management strategies promoted by Scrum and RUP is to make a “go/no-go” decision at the end of each sprint/iteration.   DAD takes this strategy further by recognizing that you have several options to consider – your team can continue with its current approach (go), it can stop (no-go), it can decide to change direction (pivot), or it can decide to experiment (run a split test).  The two new options are adopted from Lean Startup. DAD’s lean and continuous delivery lifecycles promote the idea that you make these go forward decisions when you need to, that you don’t need to wait until the end of an iteration to do so.
• Risk-oriented process goal.  The DA toolkit promotes a non-prescriptive, process goal-driven approach.  DAD explicitly includes a goal, Address Risk, the goal diagram for which is presented below in Figure 1.

Figure 1. The Address Risk process goal.

• IT risk management is built in. IT-level risk management is built into the IT Governance process blade.  Although a risk may be small for a single team, when that same risk is taken by multiple teams in parallel in aggregate the risk may be more than your organization should take on.  Hence your risk management efforts at the delivery team level must be monitored regularly and aggregate risks mitigated appropriately. The process goal diagram for IT governance is shown below in Figure 2.

Figure 2. IT-risk management is an aspect of IT Governance.

Risk management isn’t explicitly built into first generation agile methodologies like Scrum or XP, although a handful of implicit risk mitigation techniques are.  As a result you still need to develop a risk management strategy for yourself when you limit your team to these sorts of agile methods.  The DA toolkit, on the other hand, has built risk management strategies into the toolkit in a lightweight and comprehensive manner.  From the point of view of process risk, having such guidance built into your process is a low-risk and desirable approach.  For further reading about risk management, we highly suggest the list of references maintained by Glen B. Alleman.

I was recently asked how is technical debt addressed in Disciplined Agile Delivery (DAD), a very important question.  Because DAD promotes a full, explicit delivery lifecycle there are many opportunities to first avoid creating new technical debt in the first place and second to address existing technical debt appropriately.

Here are some of the strategies that DAD promotes when it pertains to technical debt:

1. Do a bit of up front thinking.  One of the process goals of DAD is Identify Architecture Strategy.  By thinking through critical technical issues before you implement your solution you have the opportunity to avoid a technical strategy which needs to be reworked at a future date.  The most effective way to deal with technical debt is to avoid it in the first place.
2. Have an explicit architecture owner.  The Architecture Owner (AO) on a disciplined agile team is responsible for guiding the team through technical decisions, particularly those at the architecture level.  AOs often mentor other team members in design skills, skills that should help them to avoid injecting new technical debt into the environment.  They should also be on the lookout for existing technical debt and where appropriate motivate the team to address that technical debt when appropriate.
3. Be enterprise aware.  Disciplined agile teams are enterprise aware, realizing that what they do should leverage and enhance the overall organizational ecosystem.  They will work close with your enterprise architecture and reuse/asset teams, if you have such, so that they can take advantage of existing assets.   Assets could include code, patterns, services, templates, guidelines, or anything else worthy of being reused.
   An important strategy for avoiding technical debt is to reuse existing assets and not rebuild or rebuy something that you already have.
4. Refactor technical debt away.  DAD provides guidance for when to apply several forms of refactoring, including code refactoring, database refactoring, and user interface (UI) refactoring.  Refactorings are typically very small, such as renaming an operation or splitting a database column, so should just be part of everyday development.  Rework, on the other hand, is more substantive and should be explicitly planned.  The Architecture owner will often negotiate rework-oriented work items with the Product Owner (the person on the team who is responsible for prioritizing the work).
5. Regression test continuously.  One of the easiest ways to find problems in your work is to have a comprehensive regression test suite that is run regularly.  This test suite will help you detect when defects are injected into your code, enabling you to fix them, or back out the changes, right away.
6. Automate code/schema analysis.  There are many tools available for assessing the quality of your code and even your database schema.  Disciplined agile teams will include the use of these tools in their continuous integration (CI) strategy.  Knowing where your technical debt exists is the first step in removing it.
7. Measure technical debt.  Organizations that are serious about technical debt measure it, something that code/schema analysis tools help with, and more importantly keep an eye on the trends (which should be going down over time).  You may choose to track code quality metrics, data quality metrics, usability metrics, time to address defects, time to add features, and many other things.
8. Explicitly govern technical debt.  Several of the previous strategies require investment that some organizations wouldn’t normally consider to be part of the mandate of a delivery team.  For your organization to succeed at reducing technical debt it must be governed, albeit in an agile fashion.  This means it needs to be understood by senior management, measured (see previous point), and funded.  The DA tool kit includes explicit guidance around how to govern agile teams effectively.
9. Reducing technical debt should be part of your culture.  Technical debt isn’t going to fix itself, and worse yet will accrue “interest” over time in the form of slower and more expensive evolution of your existing assets.
10. Address technical debt before handing over an asset.  Passing systems with high technical debt to other teams, such as a sustainment team or maintenance group is generally a bad practice. It should be ingrained in your culture that each team is responsible for keeping the quality of their solutions high. It is reasonable to expect maintenance groups to resist accepting systems that have high technical debt.
11. Accept some technical debt.  Sometimes you will decide to explicitly accept some short term technical debt for tactical reasons.  Perhaps you need to get something developed quickly because you are running a market experiment (a la Lean Startup).  Perhaps there is a new component or framework about to be delivered by another group in your organization, so you’re writing a small portion of what you need for now until you can replace it with the more robust asset.  Regardless of the reason, part of the decision to accept technical debt is to also accept the need to pay it down at some point in the future.  Having good regression testing assets in place assures that refactoring accepted technical debt in the future can be done with low risk.

There are many good online resources regarding technical debt, and the best single one that we have found is Israel Gat’s blog.   Technical debt is real and you need a viable strategy to manage it.  Otherwise you run the risk of slowly choking the life out of your organization’s IT infrastructure.  The DA too lkit can help you to understand how the strategies described above fit into your overall agile delivery process.

Related Resources

• The Spilled Juice Analogy for Technical Debt
• Data Technical Debt: How to Address Quality Problems in Data Sources
• Do Agile Teams Pay Down Technical Debt in Practice?
• Crushed by Technical Debt? (Webinar, earn 1 PDU)

A common question we get regarding Disciplined Agile (DA) is “What makes DA more disciplined than other approaches to agile?”  It’s a fair question, particularly from someone who is new to DA.  This blog posting explores this question, explicitly summarizing the critical strategies that exhibit the greater levels of discipline in DA as compared with what we see in many agile teams today.

It is clear that many common agile practices require discipline.  For example, agile teams it takes discipline to hold concise, streamlined coordination/Scrum meetings; to consistently deliver business value every iteration; to test continuously throughout the lifecycle; to improve your process “in flight”; to work closely with stakeholders and many more things.  Discipline is very important to the success of agile teams, see The Discipline of Agile for a detailed discussion, and DA takes it to the next level in the following ways:

1. Reducing the feedback cycle. Techniques that shorten the time between doing something and getting feedback about it are generally lower risk and result in lower cost to address any changes than techniques with longer feedback cycles.  Many of these techniques require agile team members to have new skills and to take a more disciplined approach to their work than they may have in less-than-agile situations.  There are several common ways to shorten the feedback cycle that are common to agile software development that are adopted by DA.  These techniques, listed in order of immediacy, include non-solo development (e.g. pair programming), active stakeholder participation, continuous integration (CI), continuous deployment (CD), short iterations/sprints, and short release cycles.
2. Continual learning. Continual learning is an important aspect of agile software development in general, not just DA.  However, DA explicitly addresses the need for three levels of learning: individual, team, and organizational/enterprise.  It also addresses the need for three categories of learning: domain, technical, and process.  Continual learning strategies include active stakeholder participation, coaching, mentoring, individual learning, non-solo development, experimenting via minimum viable products (MVPs), spikes, retrospectives/reflections, sharing lessons learned between teams, and stakeholder demonstrations.
3. Incremental delivery of consumable solutions.  Being able to deliver potentially shippable software increments at the end of each iteration is a good start that clearly requires discipline.  DA goes one step further and advises you to explicitly produce a potentially consumable solution every iteration, something that requires even greater discipline.  Every construction iteration your team requires the discipline to create working software that is “done”, to write deliverable documentation such as operations manuals and user documentation, to address consumability (usability), to consider organizational change issues pertaining to your solution, and operations and support issues (an aspect of DevOps).
4. Being process goal-driven.  The DA tool kit promotes a process goal-driven approach. For each goal we describe the issues pertaining to that the goal. For example, with initial planning you need to consider issues such as the amount of initial detail you intend to capture, the amount of ongoing detail throughout the project, the length of iterations, how you will communicate the schedule (if at all), and how you will produce an initial cost estimate (if at all). Each issue can be addressed by several strategies, each of which has trade-offs. Our experience is that this goals-driven, suggestive approach provides just enough guidance for solution delivery teams while being sufficiently flexible so that teams can tailor the process to address the context of the situation in which they find themselves in. The challenge is that it requires significant discipline by agile teams to consider the issues around each goal and then choose the strategy which that is most appropriate for them.
5. Enterprise awareness.  Whether you like it or not, as you adopt agile you will constrained by the organizational ecosystem, and you need to act accordingly.  It takes discipline to be enterprise aware and to work with enterprise folks who may not be completely agile yet, and have the patience to help them.  It takes discipline to work with your operations and support staff in a “DevOps” manner throughout the lifecycle, particularly when they may not be motivated to do so.  Despite the fact that governing bodies such as project management offices (PMOs), architecture and database authorities, and operations may indeed be a source of impediments to your DA adoption, or they may be your greatest allies, these authorities serve important functions in any large enterprise.  Therefore a disciplined approach to proactively working with them and being a positive change agent to make collaboration with them more effective is required.
6. Adopting a full delivery lifecycle.  Despite some agilists reluctance to admit that projects go through phases, DA explicitly recognizes that they do.  Building serious solutions requires a lot more than just doing the cool construction stuff.  It takes discipline to ignore this rhetoric and frame your project within the scope of a full delivery lifecycle.  The two project-based life cycles explicitly depict pre-delivery activities, a three-phase delivery lifecycle, and post-delivery activities (operations and support).
7. Streamlining inception activities.  The project life cycles include an Inception phase, known as Sprint 0 in Scrum, where you perform fundamental project initiation work.  Unfortunately in our experience we have seen many organizations treat this phase as an opportunity to do massive amounts of upfront documentation in the form of project plans, charters, and requirements specifications.  Some people have referred to the practice of doing too much transitory documentation up front on an agile project as Water-Scrum-Fall.  We cannot stress enough that this is NOT the intent of the Inception phase.  While we provide many alternatives for documenting your vision in Inception, from very heavy to very light, you should take a minimalist approach to the Inception phase and strive to reach the stakeholder consensus milestone as quickly as possible.  If you are spending more than a few weeks on this phase, you may be regressing to a Water-Scrum-Fall approach.  It takes discipline to be aware of this trap and to streamline your approach as much as possible.
8. Streamlining transition activities.  In most mid-to-large sized organizations the deployment of solutions is carefully controlled, particularly when the solutions share architectures and have project interdependencies.  For these reasons release cycles to your stakeholders are less frequent that you would like because of existing complexities within the environment.  However, the ability to frequently deploy value to your stakeholders is a competitive advantage; therefore you should reduce the release cycle as much as possible.  This requires a great degree of discipline in areas such as pre-production integration and deployment testing; regular coordination between project teams and with operations and support staff; Change management around both technology and requirements; and adoption of continuous deployment practices to such a degree that very frequent deployments are the norm and the Transition “phase” becomes an automated transition activity.
9.  Adopting agile governance. It is easier to avoid your traditional governance and tell management that “agile is different” than it is to work with your governors to adapt your governance to properly guide the delivery of your agile teams.  Every organization has a necessary degree of governance and there are ways to make it especially effective on agile initiatives.  It takes discipline to work with your governors to help them understand how disciplined agile teams operate and then discipline to accept and conform to the resulting governance process.
10. Moving to lean. For all of the process goals we describe a range of options to address the issues pertaining to that goal.  These options ranged from traditional/heavier approaches that we generally advised against except in very specific situations to agile strategies to very lean strategies.  Generally, the leaner the strategy the greater the discipline it requires.

Adopting a disciplined approach to agile delivery requires the courage to rethink some of the agile rhetoric and make compromises where necessary for the benefit of the “whole enterprise” and not just the whole team.  In our experience most agile projects make certain compromises that are not classically agile in order to get the job done.  Rather than hiding this and fearing reprisals from those who would accuse you of regressing to a traditional approach, it is better to have the courage to take a pragmatic approach to using agile in your situation.

Effective application of DA certainly requires discipline and skill, but in our experience the key determinant of success is the ability and willingness of the team to work well together and with  stakeholders, both within and external to the team.  

A common question that we get is whether it’s possible for a team to take an agile approach in a regulatory environment.  The answer of course is a resounding yes, although your approach will need to be tailored to reflect the constraints of the applicable regulation(s).

Let’s explore issues pertaining to compliance:

1. The regulations vary.  Not all regulations are created equal.  For example, financial regulations such as Sarbanes Oxley (SoX) are typically less stringent than life-critical things such as the various Federal Drug Administration (FDA) regulations.  So, one regulatory compliancy strategy does not fit all and your team will instead need to tailor their agile strategy to reflect the applicable regulations that you face.
2. Agile teams are working in a regulatory compliance scenarios.  The quick answer is yes.  As you can see in the chart above, the 2016 Agility at Scale study found that two-thirds of agile teams face either regulatory, organizational, or both forms of compliance.
3. Organizations are succeeding at applying agile within a regulatory regime.  The 2012 Agility at Scale study found that some respondents indicated that their organizations had successfully applied agile strategies with regulatory situations. As you can see in the chart below they are applying agile in all types of regulatory environments, including but not limited to life-critical and financial.  If other organizations are succeeding at doing so perhaps yours can as well.
4. Organizations are failing at this too.  The 2012 Agility at Scale study also asked if organizations had agile project teams that failed within regulatory situations and respondents indicated that they had.  If other organizations are struggling with agile and regulatory compliance then yours might too, so please consider the advice provided below.
5. The regulations rarely tell you how to work.  Regulations typically provide criteria that your process needs to meet.  For example they may call out the need to have independent testing, but they won’t say that you need to have an onerous testing phase nor that all testing needs to be done this way.  There you could adopt parallel independent testing in addition to your whole team testing efforts to conform to this requirement.  The implication is that you can tailor your solution delivery process to be as agile as you can while still being compliant – you don’t need to take a waterfall/V-model style approach.
6. Sometimes compliancy is self imposed.   Some compliancy requirements are not legislated, such as FDA and SoX, but are instead willingly adopted by your organization.  Examples of this include compliancy regimes such as ISO-900X and CMMI, strategies which may have been adopted for marketing reasons (typically by IT service providers) or perhaps process improvement reasons.  As you can see in the chart organizations are both succeeding and failing at applying agile in these situations.
7. You need to read the regulations.  Our experience is that many organizations will let their more bureaucratic-leaning staff members interpret how to conform to regulations.  Not surprisingly their strategy often involves a lot more paperwork, activities, and checkpoints than is actually needed.  When pragmatic people are asked to interpret regulations you often end up with a more pragramatic response.  So, if you’re in a regulatory environment we’ve found that it behooves you to take the time to read the regulations so that you can streamline how your agile team addresses them.  Fair warning: Most regulations are incredibly dry reading.

Disciplined Agile Delivery (DAD) addresses regulatory compliance issues via several key strategies:

1. Adopt a hybrid process.  DAD is a hybrid tool kit that adopts strategies from a range of sources including Scrum, XP, Agile Modeling, Kanban, Unified Process, and many more.  Regulations typically cover a wide range of issues and as a result you need to adopt supporting practices from numerous sources.  This may include management practices from Scrum, agile development practices from XP, agile documentation practices from Agile Modeling, data quality practices from Agile Data, and so on.  The DA toolkit has already done the heavy lifting for you by showing how these practices fit together, unlike methods such as Scrum which leave this work up to you.
2. Adopt a full delivery lifecycle.  Most regulations address the full delivery lifecycle, not just construction.  DAD supports a full delivery lifecyle, in fact it supports several such lifecycles (a Scrum-based lifecycle, a lean lifecycle, a continuous delivery lifecycle, and so on) to reflect the differing contexts faced by teams in typical enterprise environments.
3. Focus on solutions, not just software.  Disciplined agile teams produce consumable solutions, not just “shippable software”.  DAD recognizes that delivery teams are working on solutions that have a software component, that run on hardware, that are supported by documentation, and that the team may even change the business process around the usage of a system and even the organization structure of the people using it.
4. Take a goal-driven approach. Recognizing that solution delivery teams find themselves in unique situations, DAD doesn’t prescribe how they should work.  Instead, it focuses on providing advice for how teams can tailor their strategy to reflect that context of the situation that they find themselves in.  DAD does this by promoting a process goal driven approach.  This strategy guides teams through the process decisions that they’re making, some of which will be driven by regulatory compliance.  The DA tool kit has already done a lot of the heavy lifting regarding how to tailor your agile process to meeting scaling concerns such as regulatory compliance, large teams, geographically distributed teams, and other issues.  
5. Adopt an explicit governance strategy.  DAD has agile governance strategies built right in, including explicit light-weight milestones, metrics, named phases, and many other aspects of governance expected by many regulations.  Once again, DAD has done a lot of the heavy lifting for you.
6. Be enterprise aware.  DAD promotes the concept of enterprise awareness, the recognition that agile teams do not work in a vacuum.  This includes strategies for engaging with enterprise architects, how to deal with enhancement requests and defect reports coming in from operations, and how to work with other enterprise professionals.  These can be key issues to understand when tailoring agile to be compliant within an existing organizational ecosystem – your entire process needs to comply to the regulations, not just the development portion of it.

In short, yes it is possible to successfully follow a disciplined agile strategy given the constraints of regulatory compliance.