Disciplined Agile

Mark met the team that just finished the translation of the DAD book into Japanese here at IBM Innovate in Orlando.  They have created a Japanese version of the DAD Blog.  A link has been added on the main page to the Japanese blog in the Blogroll links section.  A big thank you goes out to the DAD community in Japan.  Perhaps we will run a DAD workshop there soon!

Elizabeth Woodward, co-author of one of my favorite books on scaling agile, A Practical Guide to Distributed Scrum, recently posted a video on Youtube entitled Disciplined Agile Delivery – The Shirt, the Summary .  I worked with Elizabeth at IBM when she led IBM’s internal agile educational support effort and actively worked with teams to help them to adopt and tailor disciplined agile strategies.  As you can read in her own book, she has deep expertise in successfully applying agile at scale.

In the video, which I had no idea that she was going to create, she shares her thoughts about why you should consider DAD.  She describes the value of the explicit inclusion of Inception activities, extending Scrum to describe a truly hybrid approach, and the explicit inclusion of transition activities.  The video is a little less than six minutes in length and I suspect you’ll find it a good use of your time.

As the title implies, she also has a few nice things to say about the “fabulous” DAD shirt we sent her.  Luckily, due to training from my wife, The Lovely Beverley, I knew enough to order a woman’s version for her.  Context counts!  

I was recently asked the question “What happens when the Product Owner and the Architecture Owner don’t agree?” and realized this is an issue for everyone on DAD teams in general.  Here’s my advice:

1. Talk it out.  People aren’t always going to agree, and that can often be a very good thing.  So talk it out amongst yourselves and explore why you don’t agree on an issue.  Very often someone else has a different point of view that you weren’t aware of, hence the need for a discussion.
2. Recognize that people have certain rights and responsibilities.  One of the reasons why DAD defines rights and responsibilities for various roles, which we adapted from source methods such as Scrum and Agile Modeling, is to help distinguish the decision rights of people in those roles.  For example Product Owners have the right to prioritize the work but do not have the right to dictate technical decisions.  Similarly, the Architecture Owner is responsible for guiding the team through technical decisions but does not have the right to set work priority.  An implication is that although the AO might not agree with some of the prioritization decisions being made by the PO they still need to respect those decisions.
3. Talk it out.  It’s better to talk an issue through and communicate the reasons behind a decision than to simply dictate it.

The Situation Context Framework (SCF), an evolution of the Software Development Context Framework (SDCF), defines the contextual factors to consider when selecting and tailoring a situation-dependent way of working (WoW).  The SCF is used to provide context for making decisions about how to organize your WoW to be fit-for-purpose.  Figure 1 overviews how several selection factors drive the initial choice and tailoring of your team high-level WoW, in particular your choice of lifecycle.  Of course initial selection is just the first step, you will also need to tailor your detailed choices to reflect the situation that you face - these decisions are driven by the complexity factors that you face.

Figure 1. Context factors for selection and tailoring your way of working (WoW).

Selecting an Initial WoW 

When you initiate a team you need to identify key aspects of your WoW, in particular:

1. Your team organization. When it comes to team organization you have several issues to consider.  Will the team be composed mostly of specialists such as business analysts, user experience (UX) designers, implementers and so on or will team members be more along the lines of T-skilled generalizing specialists?   How large will the team need to be?  Where will you find these people?   Will they be located in the same place or spread out?  Will they work for a single organization or several?  The choices you make will be driven by the situation that you face.
2. How you will work together. Similarly you have several process-related issues to consider.  What paradigm is most appropriate?  For example will you take an agile approach?  A lean approach?  A traditional approach?  A hybrid approach?  Will your team be able to follow a light, goal-driven process or a prescriptive one?  Will your process be constrained by compliance to frameworks such as CMMI or ISO standards?
3. What tools you’re going to use. With respect to tooling there is a myriad of options and it seems as if everyone has an opinion as to which tools are best.  However, our experience is that there are several key issues to consider when choosing tools.  Will you adopt open source tools, commercial tools, or a combination thereof?  Will your tools be integrated or stand alone?  Do you prefer to obtain tools from a single source whenever possible, with the potential for better integration and support, or will you strive for best of breed tools regardless of vendor?  Will you host your own tool environment or will it be hosted externally via a SAAS-style approach?  If hosted externally, where will your intellectual property (IP), such as source code, be hosted?

The choices that you make initially will change over time as you learn and as your situation evolves, the point is that you will make some broad choices at first to get going.

The Selection Factors

The decisions about your initial WoW will be driven by factors such as the skill and culture of the people who will potentially be on the team, your organizational culture and policies, the nature of the problem being addressed, and business constraints such as time to market and budget. Figure 2 overviews these selection factors, indicating the range of the extremes for each one - On the left-hand side is the simple extreme and on the right-hand side the challenging extreme. .

Figure 2. Selection factors.

The selection factors are:

1. Team member skills.  The people on a team must have the skills, or must gain the skills, that befit the role they play on that team.  For example, developers on an agile software team may need to have test-driven-development (TDD) skills, people-oriented collaboration/communication skills, continuous integration (CI) skills, model storming skills, team-based planning skills, and so on.  Developers on serial/traditional teams may be more focused on programming skills for a specific technology platform.
2. Team culture.  People who are collaborative and team-focused in nature are better suited for agile/lean environments whereas people who like to work alone are better suited for traditional approaches. Similarly people who are open and flexible in their approach are better suited for agile or lean strategies.
3. Organizational culture.  Your organization’s culture may vary from that of the team you are putting together, something that is particularly true when you are first learning about new ways of working.  An organizational culture that is very flexible and collaborative will mean that it is easier to take an agile or lean approach, wherease a more rigid, command-and-control culture will make it difficult to do so.
4. Nature of the problem.  Although some people want to believe that certain types of problem can only be solved in one manner that doesn’t seem to be the case in practice.  For example, it’s possible to take an agile or a traditional approach to data warehousing projects, to marketing projects, and to procuring services from an external partner.  Our experience is that the real issue is how decomposable the potential solution is.  For example, it’s possible to decompose a data warehouse into many small releases if you choose.  Same thing can be said of the development and execution of a marketing campaign.  But, it isn’t very easy to decompose an airplane into several working parts.  Either the airplane is complete or it isn’t.  Yes, it’s still possible to apply agile techniques to the building of the airplane, and very likely to it's design as well, but the airplane team will never be as agile as a software development team due to the physical and regulatory constraints that they face.
5. Business constraints.  The way that the business constrains the endeavor, such as insisting on a certain (always agressive) delivery date, an approach to managing the budget (often a fixed price/bid), and how available business people will be throughout the project certainly has an affect on the process you adopt and the type of people that you include on the team.  It may even influence what tools you use, particularly those for communication and collaboration.

The Complexity Factors Pertinent for Choosing Your WoW

The complexity factors of the SCF affect your decisions when choosing techniques/practices when you choose and evolve your WoW. Figure 3 explores these complexity factors, indicating the range of each factor.  On the left-hand side is the simple extreme and on the right-hand side the challenging extreme.  

Figure 3. Complexity factors.

Let’s examine each scaling factor one at a time:

1. Team size.  Teams can range in size from two people to twenty to two hundred or more.  Larger teams are typically formed to address more complex problems, and as a result large teams take on the challenges of greater domain complexity and/or greater technical complexity as described below.  Team size tends to directly affect how you organize the team and how you coordinate within the team.  For example, a large agile team of 600 will be organized into subteams and a leadership team will be required for coordination, something we capture in DA's Program life cycle.  A team of 50 will also be organized into subteams, although coordination will likely be simpler and possibly handled by a daily coordination meeting of representatives from each subteam (a techniques referred to as a Scrum of Scrums).  It is fairly straightforward to coordinate the activities of a team of 10 people.
2. Geographic distribution.  Agile teams may be co-located, with the team members and key stakeholders in the same room, they may be on the same floor in a single building, on multiple floors, some may work in different buildings, some may work from home, and some may even work in different countries.  A popular misconception is that agile teams need to be co-located, a misconception that I have shown via several surveys over the years to be false.  Granted, it’s a very good idea to get people working as closely as possible, but it doesn’t happen as often as we’d like.  Similar to large teams, coordination of team members throughout the project become more difficult and as a result more sophisticated coordination is required.  A greater investment in initial modeling and planning, but not much more, is required during Inception to mitigate the communication risks associated with distribution.  To increase chance of project success you will need to fly people around at key points in the project, something many organizations are loathe to do because it’s easy to measure travel costs but difficult to quantify the benefit of face-to-face collaboration.  Please read Geographically Distributed Agile Teams for a more detailed discussion.
3. Organizational distribution.  This refers to the concept of involving people from several organizations on the project.  The easiest situation to be in is to have all of your team members from the same group/division within a single organization, often the situation of a startup company or new product team within an enterprise.  It’s a little harder when people from several groups are involved.  Hiring contractors adds to the complexity.  Outsourcing a portion of the work to an external service provider is harder yet.  Partnering with several vendors even harder.  Outsourcing to one or more service providers with a very different culture than your own harder yet.  Organizationally distributed projects tend to take on the challenges associated with large teams and geographically distributed teams.  When outsourcing is involved they take on the risks associated with procurement and then the governance of the outsourced effort.
4. Skill availability. Your team needs the right people with the right skills to fulfill the outcomes that you've taken on.  At the ideal end of the spectrum you have skilled people "sitting on the bench" waiting to get going, at the other end it may take many months and potentially lots of money to build the team that you need. The availability of skilled people, or at least people with the ability to quickly gain the skills that they need, is a driver of organization distribution.  When your immediate organization doesn't have easy access to the required skills you may need to start partnering with other groups or even external organizations to gain them.
5. Compliance.  There are two forms of compliance.  Generally the simpler form of compliance is self-imposed, perhaps your organization chooses to be CMMI or ITIL compliant.  The second, and potentially harder, form of compliance is regulatory.  A team may need to conform to financial regulations, privacy regulations, or even existential/life-critical regulations.  Although every regulation has different requirements, from a process point of view they typically require extra documentation (but keep it light), reviews (keep them streamlined), and sometimes a documented process.
6. Domain complexity.  The complexity of the domain, or the problem space, being tackled by a team can vary widely.  An informational website site, such as this one, is fairly straightforward.  An e-commerce site is more difficult.  An air traffic control system even more difficult.  The greater the domain complexity that you face the more you want to invest in up-front modeling and planning.  Not much more, mind you, but still more.  Similarly as domain complexity rises it motivates greater sophistication in your agile testing strategy.  As domain complexity increases it puts a greater burden on your Product Owner, requiring more sophsticated agile modeling skills and potentially the support of agile business analysts.
7. Solution complexity.  Disicplined agile teams will face varying levels of solution complexity.  On the simple end of the spectrum you’ll have the development of a brand-new, stand-alone solutions built with new technologies.  Things get more difficult if you need to take advantage of existing assets, including software, data sources, or business services.  Things get more difficult if you need to support several technology platforms.  Things are more difficult yet if you need to refactor existing infrastructure (including legacy data sources).  As with domain complexity, the greater the solution complexity the greater the need for a bit more up-front modeling and more sophisticated testing throughout the lifecycle.  Greater techncial complexity puts a burden on your Architecture Owner, requiring great agile architecture and agile design skills of them.

History of the SCF

Where do these ideas come from?  The primary source is something called the Agile Scaling Model (ASM) which I led the development of in 2008-2009 while working for IBM.  In parallel to my work on the ASM Philippe Kruchten was working on something he calls “situational agility”, the heart of which was eight (8) factors often referred to as the “Octopus model”.  In the Autumn of 2012 Mark Lines and I began thinking about how to combine and evolve these two frameworks into one, something we originally called the Process Context Framework (PCF).  We moved away from that name because the strategy was clearly applicable to more than just software process, hence we adopted the name Software Development Context Framework (SDCF) which is inclusive of people, process, and tools.  Then of course, over the years, we applied this to far more than just software development, so we evolved this to become the Situation Context Framework (SCF) in 2020.

Related Reading

• Guided Continuous Improvement (GCI)
• The Evolve Way of Working (WoW) process blade

One of the claims that we make in the Disciplined Agile Delivery (DAD) book is that DAD provides a solid foundation from which to scale agile.  In this blog posting I thought I would expand upon that idea.

Figure 1 overviews the basic strategy that I led the development of when I was with IBM Rational.  The fundamental observation was that many organizations were struggling with how to scale agile methods, in particular Scrum.  We felt that the first step was to identify how to successfully develop a solution from end-to-end.  Although mainstream agile methods clearly provided a lot of great strategies, there really wasn’t any sort of glue beyond consultantware (e.g. hire me and I’ll show you how to do it) putting it all together.  This is where DAD comes in, but that’s only a start as you also need to tailor your approach to reflect the context in which you find yourself.

Figure 1: DAD provides a foundation for agility at scale.

First, let’s examine how DAD provides a better foundation for scaling agile:

1. Risk and value driven lifecycle.  Scrum has what is called a value driven lifecycle.  Work is prioritized by value to the business and is performed in priority order.  This is a pretty good approach, but it’s possible to do better.  Disciplined agile teams recognize that it’s a pretty good idea to tackle the riskier work early in an endeavor in order to help eliminate some or all of the risk.  Some people like to refer to this as an aspect of “failing fast” although we like to put it in terms of  succeeding early.  A risk-value approach to work prioritization, and better yet explicit risk-based milestones (such as reaching stakeholder agreement early and proving the architecture with working code early), can increase your chance of project success.
2. Self organization with effective governance.  There has been much ado made over the strategy of self organizing teams with the agile community and rightfully so as it is an effective strategy.  But, agile project teams don’t work in a vacuum but instead work within the scope and constraints of a larger, organizational ecosystem.   Instead of optimizing the project part as many agile methods imply that you should do in DAD we recommend that you adopt an effective governance strategy that guides and enables agile teams.
3. Delivery of consumable solutions over construction of working software.  There are two issues here, a delivery focus over a construction focus and a solution focus over a software focus.  First, disciplined agile teams recognize that there is some up-front project initiation/inception work that occurs early in a project.   DAD also recognizes that there is often some deployment/transition effort that occurs towards the end of a project.  The end result is that DAD promotes the idea that you need to adopt a full delivery lifecycle, not just a construction-focused lifecycle, if you’re to successfully avoid common mistakes such as a Water-Scrum-Fall approach.  Futhermore, because DAD isn’t prescriptive it suggests several versions (agile, lean, continuous delivery) of the lifecycle.  Second, agile teams do far more than produce software.  We create supporting documentation.  The software runs on hardware that may need to be upgraded and/or redeployed.  We potentially change the business process around the usage of the system we’re producing.  We may even affect changes to the organization structure of the people using the system.  In short, it is blatantly obvious that we’re not just producing “potentially shippable software” but instead are producing “potentially shippable solutions”.  Moreover, producing something that is just “potentially shippable” isn’t what our stakeholders actually want.  What they really desire is something that’s consumable, something that they can easily understand and adopt to help them achieve their goals.  The rhetoric “potentially shippable software” plays well to some developers, but it isn’t a sufficient goal.
4. Enterprise awareness over team awareness.  I alluded to this in point #2.  Disciplined agile teams recognize that they work in a larger organizational ecosystem.  This enterprise awareness  motivates them to leverage existing assets; enhance existing assets; work closely with enterprise professionals such as enterprise architects, reuse engineers, portfolio managers, and data adminstrators; and produce solutions that reflect the technology and business roadmaps of your organization.  Done right this increases a team’s ability to deliver.
5. Context-sensitive and goal driven over prescriptive.  One process size does not fit all.  It’s comfortable to think that prescriptive strategies such as managing changing requirements in the form of a product backlog, holding a daily meeting where everyone answers three questions, having a single source of requirements and thereby a neck to wring, and other ideas will get the job done.  But we all know that this isn’t true.  There are many strategies for managing requirements change, there are different ways to coordinate within a team, there are different ways to explore stakeholder needs, and so on.  Each of these strategies has advantages and disadvantages and each has a range of situations where they are appropriate.  A strategy that works for a small co-located team will put a large geographically distributed team at risk.  A strategy that works well in a non-regulatory environment may result in people’s deaths in a regulatory one (or more likely fines because hopefully you’ll be caught before you ship).  So, if you want to build an effective team you need to be able to select the right strategy for the situation you find yourself in.  DAD describes a straightforward, easy to consume strategy that is goal-driven.  This strategy has a visual component, the goals diagrams which summarize the fundamental process decision points, and a textual component (goals tables) which capture the details.

Now let’s examine what it means to scale agile.  When many people hear “scaling” they often think about large teams that may be geographically distributed in some way.  This clearly happens, and people are clearly succeeding at applying agile in these sorts of situations (see some of the more recent evidence I’ve gathered that agile scales, as well as some of the older evidence), but there’s often more to scaling than this.  Organizations are also applying agile in compliance situations, either regulatory compliance that is imposed upon them or self selected compliance (such as CMMI and ISO).  They are also applying agile in a range of problem and solution complexities, and even when multiple organizations are involved (as in outsourcing).  As Figure 1 indicates, there are several scaling factors which you need to consider when tailoring your agile strategy.

So how does DAD provide a foundation from which to scale agile?  When one considers how scaling factors can potentially affect your strategy it becomes a lot clearer.  Consider some examples:

• Geographic distribution.  When a team is geographically distributed they will likely need to to a bit more requirements envisioning up front (but not too much more), a bit more architecture envisioning up front (but not too much more), a bit more release planning up front (but not too much more), and so on.  In other words you clearly need to tailor your inception efforts.  The way the team coordinates will change (your 15 minute stand up meeting becomes one or more conference calls), the way that you coordinate requirements changes (you’re likely to have several product owners that need to negotiate dependencies), and the way that you coordinate architectural issues changes (your architecture owners will need to coordinate somehow).  In short, you need strategies that are a bit more sophisticated than having a discussion standing up around a whiteboard with some sticky notes on it.  By the way, I’ve found in several surveys over the years that the majority of agile teams are geographically distributed in some way.
• Compliance.  A few years ago I worked with an agile team that was working on FDA-compliant software.  Because of the need to be FDA compliant, they were working on a key software component of a life-critical solution, their approach to documentation, reviews, and testing was a bit more sophisticated than what you would find in a non-compliancy situation.  They needed a defined process (an early version of DAD) that met the documentation and quality constraints of FDA regulations.  This meant more documentation than most agile teams would normally create, more formal reviews, the inclusion of an independent test team on top of their whole team testing efforts, and documented proof thereof.  The point is that they were still doing documentation, reviews, and testing (amongst other activities) but doing so in a different way than if they didn’t need to be compliant.
• Technical complexity.   As technical complexity rises the sophistication of the techniques, and sometimes the tooling, needed to deal with that complexity increases.  For example, if you’re building a brand new, stand alone application your team is in a position to write clean code, create a clean UI, and create clean data storage that is fully tested from the outset.  If you’re working with a legacy system the code… may not be so clean.  It may not have a full regression test suite (making continuous integration challenging).  You may need to fix these assets, thereby requiring a more sophisticated approach to refactoring, testing, debugging, and so on than what you’re used to.  Once again, this scaling factor will affect your strategy.

The good news is that there is a growing collection of techniques for scaling agile projects.  This includes Dean Leffingwell’s Scaled Agile Framework (SAFe) as well as the continuing writings of Craig Larman and Bas Vodde.  In future blog postings we’ll discuss the scaling factors in greater detail as well as how DAD and SAFe fit together.