Disciplined Agile

Essence was created by the  Software Engineering Method and Theory (SEMAT) community and approved by The Object Management Group as a standard in 2014. The basic of Essence is that “provides the common ground for defining software development practices” (see [R1-ES]) and also is intended to build and maintain an open library and marketplace of software engineering practices and education materials (see [R3-SMMS] ).

Essence is important, (see specification [R1-ES]), because:

• “Provide a common base that is useful for software engineering endeavors of all sizes (small, medium, and large)”
• “Enable method building by the composition of practices, so that methods can be quickly assembled by a project team to match their needs, experiences, and aspirations. Allowing the method to start small and grow as needed”
• “Support method agility, so that practices and methods can be refined and modified during a project to reflect experiences, lessons learned, and changing needs”
• “Support scalability including from one product to many, from one team to many, and from one method to many”

In this article I show how to describe DAD Essentials. Essentializing DAD is different from similar endeavors because Disciplined Agile (DA) is not a prescriptive method/framework like Scrum/SAFe. Instead, DA is a toolkit based on similar goals as Essence in that it is generative – both provide building blocks from which you can tailor and evolve a process that meets your needs.

The role of this Essentials is to provide an overview of the DAD guidance and over the capabilities that could be developed for each significant aspect (“Alphas”) of an Agile & Lean process. What is really happening on Agile/Lean adoption (and in any process improvement) is developing & exercising of new capabilities. What is specific to Disciplined Agile is that it provides guidance for developing the capabilities that team & organizations needs for their specific context.

DAD Essentials are presented here in cards format using the OMG/SEMAT Essence Language constructs as Alphas, Patterns, Resources, Activities, Work Products (see Glossary section below). Each “card” has a name, a description, and a list of capabilities for your team or organization to develop.

The following terms are used in Essence.

	Alpha	An essential element that is relevant to an assessment of the progress and health of the software engineering endeavor.
	Patterns	A generic mechanism / complex concepts that are made up of several other elements
	Resources	A source of information or content.
	Activities	One or more approaches for carrying out some work to be performed and can recommend actions on alphas and/or work products in order to and relevance this work.
	Work Products	An artifact of value and relevance for a software engineering endeavor.

Full delivery life-cycles

Full, beginning-to-end, delivery life-cycle it is an explicit representation of how a software release will progress in time. The pragmatism and effectiveness of Agile (<> Waterfall) are based on realistic progress milestones with the evolution of working software toward a consumable solution.  A team could have a reference lifecycle, but in a different context, they may need to use also other types of life-cycles.

Capabilities to develop:

• A reference life-cycle
• Support for Iterative-Agile, Lean, Continuous Delivery life-cycles
• Support for high incertitude deliveries
• Support for long term roadmaps (product, business, technology)

Consumable Solution

Consumable solution is more than working software. Consumable means that we meet stakeholder needs in the context constraints and it is usable, desirable, and functional. A solution implies that, as needed, we:  

• Develop high-quality software
• Provide new or upgraded hardware/platform
• Change the business/operational processes which the stakeholders follow
• Change the organizational structure in which our stakeholders work
• Update supporting documentation

Capabilities to develop:

• Understand Consumable Solution
• Build using DAD guidance the Consumable Solution across life-cycle milestones, considering various life-cycle and practices options depending on context

Adapt to Context

DA supports two principles that motivate you to adapt your approach to your context:

• Context counts. People, teams, and organizations are all unique – leads us to a critical idea that your process and organization structure must be tailored for the situation that you currently face.
• Choice is good. Different contexts require different strategies – teams need to be able to own their own process and to experiment to discover what works in practice for them given the situation that they face. This is why the DA framework presents people with choices through the application of process goal diagrams.

Capabilities to develop:

• Understanding Agile Scaling/Tailoring Factors
• Tailor WoW in context using DA Guidance
  • Select life-cycles
  • Select practices per process goals in a given context and life-cycle
  • Use DAD Library of practices & context options guidance
  • Make experiments to see what works in your context

Core Agile Practices

Core Agile Practices will help you have a Lean process: they address the main sources of waste and have multiple benefits at the same time. It is not a coincidence that XP is based on some of them, Disciplined Agile and Agile Modeling refer them as critical practices. (See also references from Mary Poppendieck / Tom Poppendieck in “Lean Software Development”). Some core practices are:

• XP practices: Small Releases, Pair Programming, Simple Design, Refactoring, TDD, Coding Standards, and more.
• DA/Agile Modeling practices: Requirements/Architecture Envisioning, Architecture Handbook, Model Storming, Rolling Wave Planning, and many more.
• Method free practices: Clean Code/Architecture.

Interestingly, Essence describes in detail several dozen core agile practices in detail whereas Disciplined Agile puts several hundred agile, lean, and even traditional core practices into context. This is one of several reasons why Essence and DA are complementary to one another.

Capabilities to develop:

• Understand how and why you will eliminate waste
• Context usage: Core Agile Practices are not “best practices” and we still need to know the trade-offs and options for different context (See Adapt to Context alpha). Make experiments to see what works in your context

Teal Teams and Organizations

Optimize the whole: the Organization (and its constituents teams) represents that “whole” where the work optimization really makes sense. In Reinventing Organizations, Frederick Laloux presents the historical evolution of organizations from tribal to modern agile approaches:

• Red (Magic/Tribal): Impulsive, survival urgency
• Amber (Traditional/Agrarian): Authoritarian, formal hierarchy
• Orange (Scientific/Industrial): Task-oriented, profit/grow focus
• Green (Post-Modern/Information): Value-based, consensus/participative style
• Teal (Self-Organizing/Adaptive): Cellular organism with evolutionary purpose

Disciplined Agile use this model and propose as a goal the Teal organization (or at least Green): cellular, self-organizing, adaptive, aware, with evolutionary purposes. Most likely, your organization is a “rainbow” (e.g. Orange/Green/Teal). Context counts, different teams faced different situations and you can choose your strategy. You want to be at least Green because that will provide – through a participative & collaborative style – a solid foundation for further process improvement. Also, the DA Principle, Be Awesome has some expectations:

• Treat people with respect, honesty, be reliable and open
• Willingly collaborate with others

Capabilities to develop:

• Teams will offer psychological safety with clarity about roles and responsibilities
• Cross-functional skills teams, where T-skilled “generalizing specialist” is the most pragmatic, effective, efficient approach.
• Use collaborative work to envision, look ahead and just-in-time clarifications.
• Collaborative and continuous process tailoring and improvement, not only on retrospectives
• Inter-teams’ collaboration: Communities of Practices and Centers of Excellence, etc.
• Individual, team, enterprise and community level awareness
• Use pragmatic agile roles
• Address team and organization level scaling factors

Guided Process Improvement

The purpose of the Continuous Improvement process blade is to enable people within your organization to easily share their improvement learnings with one another in a systematic way. The technique of Guided Continuous Improvement (GCI) shows teams how to leverage the DA toolkit to speed up their process improvement efforts.

Capabilities to develop:

• Continuous improvement must be explicit and fundamental
• A base support for improvement should be running: life-cycles, collaborative work, retrospectives
• Kaizen strategy: continuous improvement should always be running in small steps and experiments. This lean strategy is fundamental for addressing problems complexity & incertitude.
• Guided Continuous Improvement (GCI)
  • apply the DAD toolkit to adapt to context (see corresponding alpha) for process goals as: selecting a life-cycle, forming the team, addressing changing stakeholders needs.
  • hire/listen experienced coaches
  • make progress on adopting Core Agile Practices
  • have explicit goal/guidance for Evolving your WoW

An Agile method such as Scrum, or a framework such as SAFe, can be a good start but it will have too few guidelines for choosing your way of working (WoW) in context or too little guidance for core Agile practices. DA provides the guidance for evolving your WoW and Essence the details regarding core practices. As Ivar Jacobson likes to say, this will enable you to break out of “method prison.”

Patterns

Resources

Work Products

Activities

I want to thank Scott Ambler who started this Essentializing DA initiative and collaborated with SEMAT from 2009. Scott helped me with feedback and review of current materials. DA Essentialization began with the example of the DB Refactoring technique earlier this year.

• [R1-ES] Essence Specification 1.2 – https://www.omg.org/spec/Essence/1.2/
• [R2-SM] SEMAT (Software Engineering Method and Theory) – http://semat.org/
• [R3-SMMS] SEMAT mission statement: http://semat.org/mission-statement
• [R4-ESSDB] Essentialize DB refactoring –  Database Refactoring
• [R5] DA approach for organizations – Teal is the New Black

Copyright statement

Use of Essence – Kernel and Language for Software Engineering Methods Specifications is the subject of Term and conditions & Notices found at https://www.omg.org/spec/Essence/1.2

The results of the 2018 State of Agile survey (StateOfAgile.com) have just been released.  This survey, while not particularly scientific in its approach, is a widely read and frequently quoted survey of what people are actually doing on a variety of agile topics.  It is good to see that Disciplined Agile continues to grow in popularity, up from 5% marketshare to 7%, behind only  SAFe which commands a 30% market share and is the clear leader (Scrum of Scrums is ahead of DAD, but it is just a practice, not a method).  While we are very pleased that people are finally starting to understand what DA is and how it can help them, I am not particularly fond of the way the question is framed in the survey and would like to share my thoughts for how it could be improved, and my interpretation of the findings.

1. Disciplined Agile (DAD) is listed as an option in the “Which scaling method/approach do you use?”.  People who understand DA know that it is actually not specifically a scaling framework.  It is rather a toolkit of strategies, a hybrid of practices from many methods and frameworks which can help you optimize your way of working (WoW) regardless of which approach you use.  It can be used on one small Scrum team, or dozens of SAFe teams.  Whatever approach you use, DA can help you to become even more awesome! #beawesome
2. As I said above, Scrum of Scrums should not be one of the choices as it is simply a coordination practice for Scrum at scale, not a method.
3. “Don’t know” is interesting as an option.  It puzzles me that people that are answering this survey aren’t aware of their organization’s approach.  My suspicion is that many of the people picking this selection actually mean “Not applicable” as many organizations do not scale agile.  I think that this should available as a selection.
4. The question really should be a multiple choice, rather than single.  Most organizations use a variety of approaches.  It would be more useful to ask “What percentage of your IT spend uses each of the following approaches?”
5. Spotify is actually not a framework.  It is how a Swedish music company circa 2014 had adapted agile at that point in time, to optimize their WoW for their unique context.  If you copy their approach you are copying an old approach of a company in a situation unlike yourselves, and for which they have evolved away from significantly.
6. I find “Internally created methods” intriguing as a choice.  We think that this is what all companies should aspire towards.  Start with either where you are currently, or one of the other methods (recipes), and then use the DA Toolkit to either evolve away from, or to improve your approach for your unique organization and team context.
7. Spotify actually embodies this approach.  They have continually evolved, improving, and optimizing their way of working.  Menlo Innovations also has done the same thing, starting with Extreme Programming (XP) as their core method, and then optimized for what works for them.  Rather than copying other companies approaches we should “learn how to learn” about what works best for us. We describe this approach of leveraging proven fit-for-context practices in our Choose your Wow! book as “Continuous Guided Improvement”.  Starting with some basic scaffolding of an existing method (what we refer to as “lifecycles” in DAD) provides a jumpstart on your WoW optimization.

We would recommend that you do not aspire to “be Spotify”, but rather “be like Spotify”.  Start with a basic method/lifecycle (recipe), then spice it up with the help of proven strategies from the DA Toolkit (ingredients).

Become your own Spotify or Menlo, not somebody else’s.

Thoughts?

Please don’t call yourself a “Disciplined Agile shop”.  It is the kiss of death.  I was recently having a discussion with a client about visiting them on an upcoming visit to the UK.  At one point in the past they had proudly declared themselves a Disciplined Agile “shop”.  The senior exec that I spoke with told me “We have moved on now from DA to SAFe”.  Upon further discussion he admitted that SAFe is being used on just a few initiatives, but that they had sent lots of people to SAFe training.  The inference seemed to be that since they had previously taken DA training, but had now taken SAFe training that they have now changed “shops”.  Ironically while a lot of budget was committed to SAFe training and related consulting, most of the work done at this organization actually used other agile and lean lifecycles such as Scrum and Kanban.  You know, DAD stuff.

Why is it that our industry has an obsession with labelling themselves as a type of “shop”, when the reality is that they will likely use a variety of approaches depending on the context? You could be building something from scratch, extending a solution, or implementing a commercial off the self package.  You could be in a straightforward situation, or building defence or life critical systems.  Our industry has become extremely fragmented, with organizations trying to put themselves in a certain box such as Scrum, SAFe, Scrum/XP, LeSS, DAD, Kanban, Spotify, and it goes on and on.  So let’s stop doing this.

Unfortunately, Disciplined Agile (aka DAD) has gotten lumped in with scaling frameworks such as SAFE, LeSS, Nexus etc, when the reality is that DA is not a purpose-built framework for scaling situations exclusively, like a true scaling framework.  It is, rather, a rich and flexible toolkit than can be used to apply fit-for-context strategies for your unique situation for initiatives of all sizes and types.  If you need to apply them at scale, you can.  But our preferred approach is to descale where possible rather than apply a prescriptive recipe to a large and risky problem.

When we take a closer look at different types of shops, we see a lot of MethodBut.  For example, ScrumBut is where teams use Scrum but they don’t do retrospectives or some other ceremony.  Or teams use SAFe but management doesn’t buy in to doing quarterly 2-day big room planning sessions.  Practitioners in these “shops” are ostracized and may be excommunicated from their religion for not following one of their prescribed ceremonies.  Disciplined Agile’s core principles include context counts, choice is good and pragmatism.  While we believe that skipping parts of a method early in your adoption is likely a mistake, as your teams mature and understand options that can make them more efficient we support the idea of being freed from the “method prisons” (as described by Ivar Jacobson) so that they can optimize their WoW (Way of Working).  This essentially is why the DA “toolkit” was created.  It contains hundreds of strategies to help you to make better decisions on your journey to high performance agility.

As you can see from the diagram, regardless of what framework or method you are using, there will likely be strategies that supplement your approach which are not described in the method recipe(s) that you have chosen.  DA is a toolkit of ingredients, to enable you to be a better chef.  If you don’t know what is in the pantry, and which combinations will delight the unique preferences of your guests/stakeholders, then you probably won’t meet your potential as a Michelin Star Chef.

We have come to realize that the methods/framework industry is a moving target.  Waterfall shops, then RUP, then SAFe, then….?  There will be more frameworks, indeed it seems that we learn of a new one every few months.  As consultants seek to differentiate themselves, and have something new to sell, or organizations fail in their agile adoption and look for the “next big thing”, new frameworks/recipes will continue to emerge, with related training programs and certifications.

Regardless of the recipe, the main ingredients for them don’t change that much.  Yes, new ingredients emerge, such as mob programming, UX practices, WSJF, etc.  But generally accepted fit-for-context principles tend to last.  Disciplined Agile is an agnostic approach to solution delivery.  A rich toolkit to help you to make better decisions, leading to better outcomes.

At Disciplined Agile, we are beginning to make a concerted effort to separate ourselves from the toolkits, as we really shouldn’t be competing with them.  It is not DA or .  It should be DA and .  So our recommendation is to take a DA workshop to expand your pantry of ingredients, and learn how to be a better cook. Or a good starting point is to read the Choose your Wow! book (Ambler/Lines).  But please, don’t call yourselves a “Disciplined Agile shop”.

In my experience in running dozens of process tailoring workshops, over several years, with of teams of every shape size and experience level and in different organizations, the most recurring comment is that the workshops “revealed all kinds of options we didn’t even realize were options!”   Although almost always a bit of a hard sell at the outset, I have yet to work with a team unable to quickly grasp and appreciate the value of these activities.

I had to do quite a bit of experimenting in order to get the timing and content of the workshops right – and learned over time that success is also predicated on knowing whom to include when. My first attempts were gruelling, close-to-full day affairs with entire teams in attendance, held at or close to project kickoff. Though transparent and inclusive, to my surprise this approach was actually deemed a waste of their time by many team members, especially those whose contribution would occur primarily in the construction phase. First lesson learned – A technical team lead, architect or senior developer can actually stand in for most of the developers in the early stages. I find it helpful to always bear in mind what George Dinwiddie (http://www.velocitypartners.net/blog/2014/02/11/the-3-amigos-in-agile-teams/) dubbed “the 3 amigos” in determining who needs to attend a process tailoring session. Be it at inception, construction, or even in transition, you need to tailor not only the processes, but also the attendance of the workshop in order to ensure you have the right mix of people, with the right collaborative mindset, to cover issues pertaining to 1) the business problem being addressed  2) the potential technical solutions to that business problem and 3) the processes (both team and organizational) that will enable the work to be carried out.

My second lesson learned pertained to the format and presentation of the process blades themselves. I found that simply working from the published process maps was insufficient, as we ran into onerous issues around how to best record the WoW choices teams were making. I eventually reproduced the entire process blade library in a spreadsheet format, with columns for comments. This seemingly innocuous administrative step quickly ushered in the third lesson learned – the sessions can be used not merely to document an immediate WoW decision, but also to identify future, more “mature” aspirational choices which the team can set as goals over a specified time period.

A fourth lesson learned, and one that was also enabled by using a simple spreadsheet tool, is that it became far easier to Align with Enterprise Direction. By “locking down” enterprise-level process choices across all the blades where applicable, a lot of potentially fruitless (at that point in time) discussion was saved for many a team. No use in discussing test automation strategies to death for instance in divisions still completely relying on manual tests, or at the opposite end of the spectrum, teams endowed with high-performing, well-integrated CI/CD environments. This is a large part of what DA calls self-organization within appropriate governance.

The fifth and final major lesson learned was to never start from a blank slate if at all possible. I would typically show up at a team’s first process tailoring workshop with a pre-filled version from another team facing somewhat similar challenges (the identifying data being scrubbed so they could not identify the previous team). I would then challenge the new team to reflect on the choices and determine whether they made sense for their own context. This also saved time and effort, as there are recurring themes and common challenges within organizations that all teams face.

Here’s an important note on determining participation – Ultimately, the teams themselves are the best arbiters of who should attend the sessions at varying stages of advancement. Allowing this will typically result in a bit of initial over participation, followed by under participation (especially is the pressure is on to get “real” work done!) – the key as facilitator is to coax the team back into balanced participation, and to lobby the organization for the necessary support in freeing people up. The support will become easier and easier to obtain as the benefits of allowing teams to choose their WoW become apparent.

Finally, be prepared for surprises. I once ran through the Program process blade with a team, only to have them come to the realization that … they weren’t really a Program! Which was actually a good thing as it helped avoid introducing a considerable amount of overhead, particularly in the area of program-level KPIs.

What is the point of the retrospective?

The retrospective is one of the most important ceremonies in all of agile.  This is the time the team spends together to assess how they are working together and define steps to improve that process.  It needs to be a “safe place” where people are able to speak openly and honestly.  This is their opportunity to air their dirty laundry and work through their inter-personal issues.  This is a time of growth for the team and for the team to take ownership of improvement.  The team lead will facilitating the retrospective and should manage the interactions to keep the environment safe.

Define the Team

If the retrospective is a team ceremony, then what do we mean by team?   The team includes: the team lead, the architecture owner and all other members that actively contributed to meeting the deliverables for the iteration.  This includes: developers, testers, BAs, QAs, or other specialists such as technical writers, database engineers etc.

What about the Product Owner (PO)?

The PO is NOT a member of the team.  They certainly interact with the team but they do not contribute to meeting the deliverables for the iteration so they are not a member of the team.  They are not allowed to participate in the planning poker for the user stories for the same reason.  The team votes because they are on the hook for delivering based on the sizing and the estimates but the PO is not on the hook, so they don’t get a vote.

Should the Product Owner participate in the retrospective?

In general, I would start by including the PO in the retrospectives because the team does have to learn and adjust to working with the PO. Keep in mind though that the PO may come and go but the team should stay together so it is most important that the team works well together.??As a coach, I usually talk to the PO beforehand to say that they are an invited guest and that it is a privilege to be part of the meeting so they should act accordingly.   I have been in many situations where the PO was welcomed at the retrospective, and felt left out if not included.  I favor building trust between the business (the PO is their representative) and IT (the team).  Including the PO in the retrospective can help the PO assimilate with the team.

I have also had several situations where as the coach I had to ban the PO from the retrospective because they were too commanding and disruptive in the meeting for the team to have an effective retrospective.  I have also seen many situations where the PO is also the resource manager of members of the team (which in itself is not recommended).  Having managers in the room can definitely have a dampening effect on the member’s willingness to be open and honest about problems and solutions.

If the PO doesn’t participate, at least as an observer, the team runs the risk of having to “sell” the cost of their improvement actions (against other backlog items) after coming up with them. Hopefully the PO is engaged enough with the team to understand its weaknesses and support improvement in those areas whether then attend the meetings or not.

Team Decision

Retrospectives are about improving the process, and a non-trivial part of that is optimization of collaboration between the PO and the team.  I would suggest that the team should decide whether or not to include the Product owner.

What about the Stakeholders?

The retrospective should absolutely be a closed door session for the stakeholders since the retrospective must be a “safe space”.

There was a twitter debate lately that talked about a team being subjected to “a drive by criticism from 2 PM’s during a Retrospective”.  This is a good reminder why we constrain attendance.  The “safe place” is affected by the presence of people with positional authority, potential agendas or other implicit impact.??The team may decide to invite such people – usually to ensure that they are communicating improvements needed that are beyond their locus of control.??Having outsiders as guests at the retrospective will change the dynamics but at least it is a team decision to do so.

It is very important that the team own their process.  If they’re uncomfortable that someone is in the room then that person should be asked to either change their behavior or leave (perhaps to be invited back in the future).??The coach should always be thinking along the lines of “do we have the right people in the room” and then act accordingly

Isn’t agile all about transparency?

There was a twitter debate lately that centered on transparency.  I believe that transparency is a key element to making agile successful.   I’m all for transparency in everything about agile; EXCEPT the retrospective!??Sometimes you need to have a family meeting outside of the public eye and that is the retrospective. The retrospective is all about resolving your issues in private so that you can present a united front to the rest of the world. To use a sports analogy, an NHL coach doesn’t invite the business (fans) into the dressing room between periods.  There are lots of other places for transparency; the retrospective doesn’t have to be one of them.

The output of the Retrospective

While the actual retrospective meeting is closed to other observers, I would suggest that the action items coming out of the retrospective need to be made public and posted as an information radiator for everyone to see.  The changes are more likely to get implemented if the team sees them every day.  The team may also want to “radiate” their improvement actions on their dashboard.

The actions and results of the actions may also be shared with other teams through what is often called a Retrospective of Retrospectives. I encourage teams to only choose one or two areas for improvement at a time to provide focus and make meaningful progress.