Disciplined Agile

On Saturday, February 9 a group of people got together in Whistler British Columbia to discuss the importance of context as it pertains to software development.  The workshop was organized by the good folk at Software Development Experts (once again, thank you very much).  We shared some very important insights and had some very informative discussions.

The morning began with a kick-off from Carson Holmes.  I gave a talk summarizing the work around context frameworks, including Philippe Kruchten’s Situational Agility and the work I did at IBM around the Agile Scaling Model.  My talk summarized what we’re calling the Process Context Framework (PCF), something I will blog about soon.  Philippe then summarized his recent work on cognitive biases which explains why people, including software professionals, make suboptimal decisions and more importantly how we can avoid doing so.  Mark Kennaley overviewed the theory behind Software Development Practice Advisor and demoed some of its capabilities.  This is a product that I’ve been very interested in for some time as it’s an expert system that supports better process-oriented decisions around software development that is based on true empiricism.  Steve Adolph shared his work around how software development teams collaborate successfully, including empirical observations from is grounded-theory based work for his PHd.  Julian Holmes led us through an exercise around the contextualization of 400 agile/lean practices.  Mark and Carson ended the workshop by leading the group through a discussion of how we can work together effectively.

We had very interesting discussions throughout the day.  Disciplined Agile Delivery (DAD) was discussed at several points throughout the day.  In my presentation I briefly overviewed DAD’s goal-driven approach as I believe it is a perfect example of how to instantiate contextualization of software development.  Mark Kennaley discussed how DAD is supported as a first-class citizen in Advisor in his presentation.  For the handful of people who didn’t yet have the DAD book we gave them copies signed by Mark Lines and myself.

On Sunday many of us hit the slopes at Blackcombe/Whistler.  Below is a picture of several of us at near the top of Whistler.  More of the group went skiing than just the people appearing in the picture.

The Workshop Attendees

Back row:

• Adam Murray (A.R. Murray)
• Carson Holmes (Software Development Experts)
• Steve Adolph

Middle row:

• Sheynal Saujani 
• Doug Stewart (Blueprint)
• Mark Lines 
• Ricardo Garcia (Costco)
• Arun Zachariah (Capgemini)
• Rolf Reitzig (Ascendent Technology)
• Scott Ambler 
• Philippe Kruchten (University of British Columbia)
• Chris Armstrong (Armstrong Process Group)
• Mark Kennaley (Software Development Experts)

Front row:

• Julian Holmes (Agilementors)
• Cecile Peraire (Carnegie Mellon University)
• Geri Schneider-Winters (Wyyzzk)
• Tom Weinberger (Citicorp)
• Hadar Ziv
• Mike Bonar

Enterprise awareness is one of the key aspects of the Disciplined Agile (DA) toolkit.  The observation is that DAD teams work within your organization’s enterprise ecosystem, as do all other teams.  There are often existing systems currently in production and minimally your solution shouldn’t impact them.  Better yet your solution will hopefully leverage existing functionality and data available in production.   You will often have other teams working in parallel to your team, and you may wish to take advantage of a portion of what they’re doing and vice versa.  Your organization may be working towards business or technical visions which your team should contribute to.  A governance strategy exists which hopefully enhances what your team is doing.

What it Means to be Enterprise Aware

Enterprise awareness is an important aspect of self discipline because as a professional you should strive to do what’s right for your organization and not just what’s interesting for you. Teams developing in isolation may choose to build something from scratch, or use different development tools, or create different data sources, when perfectly good ones that have been successfully installed, tested, configured, and fine-tuned already exist within the organization.  Disciplined agile professionals will:

• Work closely with enterprise professionals.  This includes working closely with enterprise technical architects and reuse engineers to leverage and enhance the existing and “to be” technical infrastructure; enterprise business architects and portfolio managers to fit into the overall business ecosystem; senior managers who should be governing the various teams appropriately; operations staff to support your organization’s overall development and operations (DevOps) efforts; data administrators to access and improve existing data sources; IT development support people to understand and follow enterprise IT guidance; and business experts who share their market insights, sales forecasts, service forecasts, and other important concerns.  In other words, DAD teams should adopt what Mark refers to as a “whole enterprise” mindset.
• Adopt and follow enterprise guidance.  Your organization may have, or hopes to one day have, a range of standards and guidelines (guidance) that it wants delivery teams to adopt and follow.  This may include guidance for coding, user interface development, security, and data conventions to name a few.  Following common guidance increases the consistency and maintainability of your solutions, and thus your overall quality.
• Leverage enterprise assets. There may be many enterprise assets, or at least there should be, which you can use and evolve.  DAD teams strive to work to a common infrastructure; for example, they use the enterprise-approved technologies and data sources whenever possible, and better yet they work to the “to be” vision for your infrastructure.  If your organization uses a disciplined architecture-centric approach to building enterprise software, there will be a growing library of service-based components to reuse and improve upon for the benefit of all current and future solutions.  To do this DAD teams will collaborate with enterprise professionals throughout the lifecycle and particularly during Inception during envisioning efforts.   Figure 1 summarizes the Inception phase goal Align with Enterprise Direction which summarizes the strategies you may choose to follow.  Read Disciplined Agilists Take a Goal-Driven Approach for more information on DAD’s goal-driven strategy.

Figure 1. Inception Goal: Align with Enterprise Direction.

• Enhance your organizational ecosystem. The solution being delivered by a DAD team should minimally fit into the existing organizational ecosystem – the business processes and systems supporting them – it should better yet enhance that ecosystem.  To do this, the first step is to leverage existing enterprise assets wherever possible as described above, often working with enterprise architects to do so. In addition to the enterprise architects DAD teams will also work with operations and support staff closely throughout the lifecycle to ensure that they understand the current state and direction of the organizational ecosystem.  DAD teams will often be supported by an additional independent test team that will perform production integration testing (amongst other things) to ensure that your solution works within the target production environment which it will face at deployment time.  Furthermore, experienced DAD teams will even fix problems that they run into via proven refactoring techniques.  Figure 2 summarizes the general goal Leverage and Enhance Existing Infrastructure which summarizes strategies for how DAD teams may accomplish this.

Figure 2. General Goal: Leverage and Enhance Existing Infrastructure.

• Adopt a DevOps Culture. DAD teams will work with operations and support staff closely throughout the lifecycle, particularly the closer you get to releasing into production.  DevOps culture and strategies are baked right into DAD, a topic for a future blog posting.
• Share learnings.  DAD teams are learning oriented, and one way to learn is to hear about the experiences of others.  The implication is that DAD teams must also be prepared to share their own learnings with other teams.  To do this organizations might choose to support agile discussion forums, informal presentations, training sessions delivered by senior team members, and internal conferences to name a few strategies.
• Adopt appropriate governance strategies.  Effective governance strategies should enhance that which is being governed. An appropriate approach to governing agile delivery projects, and we suspect other types of efforts, is based on motivating and then enabling people to do what is right for your organization. What is right will of course vary, but this typically includes motivating teams to take advantage of, and to evolve, existing corporate assets following common guidelines to increase consistency, and working towards a shared vision for your organization. Appropriate governance is based on trust and collaboration. Appropriate governance strategies should enhance the ability of DAD teams to deliver business value to their stakeholders in a cost effective and timely manner.  Unfortunately many existing IT governance strategies are based on a command-and-control, bureaucratic approach which often proves ineffective in practice. The DAD book explores appropriate governance, the impact of traditional governance strategies, and how to adopt an appropriate governance strategy in detail.  The article Adopting Agile Governance Requires Discipline also provides greater insight.
• Open and honest monitoring. Although agile approaches are based on trust, smart governance strategies are based on a “trust but verify and then guide” mindset.  An important aspect of appropriate governance is the monitoring of project teams through various means.  One strategy is for anyone interested in the current status of a DAD project team to attend their daily coordination meeting and listen in, a strategy promoted by the Scrum community.  Although it’s a great strategy we highly recommend, it unfortunately doesn’t scale very well because the senior managers responsible for governance are often busy people with many efforts to govern, not just your team.  Hence the need for more sophisticated strategies such as an “development intelligence” approach supported via automated dashboards.  More on this in a future blog posting too.

Why is Enterprise Awareness Important?

Enterprise awareness is important for several reasons.  First, you can reduce overall delivery time and cost by leveraging existing assets.  In other words, DAD teams can spent less time reinventing the wheel and more time producing real value for their stakeholders.  Second, by working closely with enterprise professionals DAD teams can get going in the right direction easily.  Third, it increases the chance that your delivery team will help to optimize the organizational whole, and not just the “solution part” that it is tasked to work on.  As the lean software development movement aptly shows this increases team effectiveness by reducing time to market.

Challenges to Enterprise Awareness

Unfortunately there are two main challenges to supporting enterprise awareness on agile teams.  First is the cultural challenge within the agile community that some “agile purists” perceive this as unecessary overhead.  Reasons for this misunderstanding include a lack of understanding of the overall enterprise picture or some agilists who have previous experiences with enterprise professionals who struggle to work in an agile manner.  This points to the second challenge that enterprise professionals often don’t understand how to work effectively with agile teams.  Sometimes this is because the agile teams they’ve been working with until now haven’t been sufficiently disciplined to work with them effectively, but more often than not it’s because they still choose to follow older, more traditional approaches to their craft (they may find my articles about Agile Enterprise Architecture, Agile Enterprise Administration, and even The Enterprise Unified Process to be illuminating).

These challenges are cultural in nature, and thus difficult to overcome.  Agilists and enterprise professionals need to respect one another and strive to learn more about what the other group is trying to accomplish.  They must strive to work with one another and thus learn from each other.  Furthermore, they must build a culture of shared commitment and responsibility to the organization.  Not only is this possible it is highly desirable.

In summary, DAD teams and more importantly DAD practitioners are enterprise aware.  They recognize that enterprise aware strategies improve their ability to provide value to their stakeholders both within the scope of a solution as well as at the organizational level.  To coin an environmental cliché: Disciplined agilists act locally and think globally.

Material for this blog posting was modified from Discplined Agile Delivery: A Practitioner’s Guide to Agile Software Delivery in the Enterprise by Scott W. Ambler and Mark Lines (IBM Press, 2012)

In this posting we explore the goal-driven aspect of the Disciplined Agile (DA) toolkit.   This goal-driven approach enables DAD to avoid being prescriptive and thereby be more flexible and easier to scale than other agile methods.  For example, where Scrum prescribes a value-driven Product Backlog approach to managing requirements DAD instead says that during construction you have the goal of addressing changing stakeholder needs.  DAD also indicates that there are several process factors/issues surrounding that goal that you need to consider, and there are several techniques/practices that you should consider adopting to do so.  DAD goes further and describes the advantages and disadvantages of each technique and in what situations it is best suited for.  Yes, Scrum’s Product Backlog approach is one way to address changing stakeholder needs but it isn’t the only option nor is it the best option in most situations.

We start by describing how to visualize goals.  We then summarize the goals called out by DAD, a topic we’ve written about in the past so we only cover this topic briefly here.  We end with a summary of the advantages and disadvantages of a goal-driven approach over the more prescriptive approaches of older agile methods.

Visualizing Goals

In the original DAD book we described process goals in a non-visual manner using tables which explored the advantages and disadvantages of the techniques associated with a process factor.  Since we wrote that book both Mark and I have spent a lot of time helping people to understand what a goals-driven approach entails and we’ve found that many people respond well to visual representations of a process goal.  Yes, the process decision tables are very important but a visual overview helps to provide context for the detailed information.

In the second half of 2012 we began developing a way to represent goals in a visual manner using what we call a goals diagram.  A goals diagram, the notation for which is summarized in Figure 1, is in effect a form of decision tree.  In Figure 1 you see that a process goal is indicated using a rounded rectangle and the decision points pertaining to a goal with normal rectangles.  Process goals will have one or more decision points that you need to consider addressing, with most goals having four or five decision points although some have eight or nine.  Each decision point is then addressed by two or more techniques/practices.   Because there may be many techniques to choose from, we indicate “default” techniques in bolded italics.  These defaults are good starting points for teams new to agile – they are almost always strategies from Scrum, XP, or Agile Modelling with a few Rational Unified Process (RUP) ideas thrown in to round things out.  Some decision points you may choose not to address.  Sometimes options are “ordered”, which is indicated by a upwards pointing arrow to the left of the list of techniques.  What we mean by this is that the techniques appearing at the top of the list are more desirable from the point of view of agile and lean thinking and the less desirable techniques are at the bottom of the stack.  Your team of course should strive to adopt the most effective techniques they are capable of performing given the context of the situation that they face.  In Figure 1 the first decision point has an ordered set of options whereas the second one does not.  Typically when the options are ordered you will only choose one of them whereas you MIGHT choose several options in unordered situations.

Figure 1. The notation of goal diagrams.

Let’s work through some examples.  Figure 2 depicts the goal diagram for Explore Initial Scope, a goal that you should address at the beginning of a project during the Inception phase (remember, DAD promotes a full delivery lifecycle, not just a construction lifecycle).  Where some agile methods will simply advise you to populate your product backlog with some initial user stories the goal diagram of Figure 2 makes it clear that you might want to be a bit more sophisticated in your approach.  What level of detail should you capture, if any (a light specification approach of writing up some index cards and a few whiteboard sketches is just one option you should consider)?  What view types should you consider (user stories are one approach to usage modeling, but shouldn’t you consider other views to explore the data or the UI)?  Notice how we suggest that you likely want to default to capturing usage in some way, basic domain concepts (e.g. via a high-level conceptual diagram) in some way, and non-functional requirements in some way.  There are different strategies you may want to consider for going about modeling.  You should also start thinking about your approach to managing your work.  In DAD we make it clear that agile teams do more than just implement new requirements, hence our recommendation to default to a work item stack over Scrum’s simplistic Product Backlog strategy.  Finally Figure 2 makes it clear that when you’re exploring the initial scope of your effort that you should capture non-functional requirements – such as reliability, availability, and security requirements (among many) – in some manner.

Figure 2. Exploring the initial scope.

Figure 3 depicts one of the goals that you should address during the construction phase, in this case Address Changing Stakeholder Needs.  This is an iteresting example for two reasons.  First, it captures the key decisions surrounding the second of the 15 principles  of the Disciplined Agile Manifesto, that of welcoming changing requirements.  Second, it has a decision point that overlaps with that of another goal, in this case we indicate that your Work Item Management Strategy is important to consider for both this goal and Explore Initial Scope (see Figure 2).

Figure 3 makes the process factors surrounding how to address changing stakeholder needs very explicit.  How are you going to prioritize changes?  A business value approach is one option, the approach popularized by Scrum, but we’ve found that the risk-value approach promoted by Unified Process (UP) to be a more robust strategy that leads to greater chance of agile project success.  There’s advantages and disadvantages to each technique so you’ll want to choose the one best for you.  When are you going to accept the change?  During the current iteration as Extreme Programming (XP) suggests or a future iteration as Scrum suggests?  Do changes come directly from stakeholders or via a proxy such as a product owner or business analyst?  How will your team elicit changes (via modeling, demos, …)?

Figure 3. Addressing changing stakeholder needs.

The advantage of visualizing goals as we’ve shown in Figures 2 and 3 is that it makes it very clear what process-related decisions you need to make and what options you have available to you. The disadvantage of this sort of diagram is that they get fairly big at times, as you can see.  This effectively prevents us from taking the diagrams one step further to indicate the trade-offs associated with each technique and as a result you’ll still need the text tables we included in the DAD book for that.

The Goals of DAD

In the previous section we indicated that there are many goals called out by DAD,  Figure 4 summarizes these goals, which have evolved slightly from what we published in the book (we refactored a few to make them more consumable).  Notice how each of the three phases (Inception, Construction, and Transition) are described by specific goals.  Also notice how some goals, such as Grow Team Members and Address Risk, are applicable throughout the entire lifecycle.

Figure 4. Goals throughout the lifecycle.

Mark Lines’ post Being Goal-Driven Requires Discipline explores the history of the figure above if you’re interested in how the goals have evolved since the DAD book was published.

The Advantage of Goals Over Prescription

First and foremost, DAD is a process decision framework.   One what that it achieves this through it’s goal-driven approach that guides people through the process-related decisions that they need to make to tailor and scale agile strategies to address the context of the situation that they face.  Our experience is that there are several fundamental advantages to taking a goal driven approach to agile solution delivery.  A goal-driven approach:

1. Supports process tailoring. I think that Figures 2 and 3 make it very clear how DAD enables people to make intelligent process decisions.  I think that this is a huge improvement over previous process frameworks, particularly Rational Unified Process (RUP) that provided a lot of great process advice (regardless of what some agilists may claim) but struggled to provide consumable process tailoring advice.
2. Enables effective scaling.  DAD provides a foundation from which to scale agile approaches.  An important part of scaling agile is to tailor your strategy to reflect the realities of the scaling factors which you face.  For example, consider your approach to exploring the initial scope of your effort (the goal captured in Figure 2).  A large agile team or a geographically distributed team will make different tailoring decisions than a small co-located team.  A team in a regulatory environment will make different decisions, particularly around amount of detail, than teams in non-regulatory environments.  These are just three of several scaling factors (more on this in a future blog posting, although you may postings in my agility at scale blog to be of interest).
3. Makes your process options very clear.  Figure 4, in combination with the more detailed goals diagrams (such as in Figures 2 and 3) make it very clear what you need to consider when tailoring an agile solution delivery process to meet the unique needs of the situation faced by your team.
4. Takes the guesswork out of extending agile methods.  Although it makes for wonderful marketing rhetoric, it’s disingenuous for people to claim that simple methods such as Scrum can be tailored to meet your actual needs.  Yes, we suppose this claim is true but how do you do so?  Shouldn’t you start with a full delivery lifecycle, not just a construction lifecycle? Shouldn’t the toolkit cover a wider range of issues, such as leadership and requirements management as Scrum does, technical issues as XP does, modeling and documentation as Agile Modeling does, and many other issues?  In short, shouldn’t it be a hybrid?   Finally, shouldn’t you be given some context-sensitive advice for tailoring the details, as we do with the goal-driven approach described here?
5. Makes it clear what risks you’re taking on.  By making your process decision options clear, and by describing the trade-offs associated with those options, DAD makes it very clear what risks you’re taking on.  Want to write a detailed requirement specification up front (yes, in a very small number of situations this is in fact a viable option for agile teams) then DAD is going to make it very clear what risks you’ve just taken on by doing so.  DAD also makes it clear when this decision is appropriate, so if you’re not in this situation then it  is likely time to rethink your approach.  Although we cannot prevent challenges such as a Water-Scrum-Fall approach where a heavy approach is taken to Inception and Transition and an agile/Scrum approach to Construction we can certainly make it very clear what the impact is of the decisions that led you to that approach.  Since the first DAD book came out in June 2012 we’ve spoken with several people who have used the decision tables in it to argue against inappropriate process decisions on their projects.  In many situations the argument “that isn’t agile” falls on deaf ears, whereas “that will take longer and here’s why”, “that will be more expensive and here’s why”, “that will result in lower stakholder value and here’s why” will be listened to.
6. It hints at an agile maturity model.  We have written about how DAD and CMMI potentially fit together.  In that article I suggested that in the case of issues where the options are ordered there is a clearly an indication of agile maturity or sophistication.  Having said that I have no desire to wade into the agile maturity model morass, but I think it’s an important observation nonetheless.

So far we’ve identified two disadvantages to DAD’s goal-driven approach when working with customer organizations.  First, it makes the complexities of solution delivery explicit.  Although some of us want to believe that the simplistic strategies of other agile methods will get the job done we inherently know that software development, or more accurately solution delivery, is in fact a complex endeavor in practice.  Second, some people just want to be told what to do and actually prefer a prescriptive approach.  DAD mitigates this problem a bit by suggesting default starting points (shown in italized bold text in the goal diagrams) but even this can be overwhelming for some people.  Interestingly, when we were writing the book two of our 30+ reviewers were adamantly against giving people choices because they felt it was better to adopt a more prescriptive approach as we see in older agile methods.

We hope that this blog posting has given you some food for thought that you can leverage on your next agile project.  Got Discipline?

Mark was interviewed on DAD at the IBM Innovate conference:

http://www.youtube.com/watch?feature=player_embedded&v=lkDBs1Gf4TA

This blog has been replaced with the page Full Agile Delivery Lifecycles.