Disciplined Agile

Essence was created by the  Software Engineering Method and Theory (SEMAT) community and approved by The Object Management Group as a standard in 2014. The basic of Essence is that “provides the common ground for defining software development practices” (see [R1-ES]) and also is intended to build and maintain an open library and marketplace of software engineering practices and education materials (see [R3-SMMS] ).

Essence is important, (see specification [R1-ES]), because:

• “Provide a common base that is useful for software engineering endeavors of all sizes (small, medium, and large)”
• “Enable method building by the composition of practices, so that methods can be quickly assembled by a project team to match their needs, experiences, and aspirations. Allowing the method to start small and grow as needed”
• “Support method agility, so that practices and methods can be refined and modified during a project to reflect experiences, lessons learned, and changing needs”
• “Support scalability including from one product to many, from one team to many, and from one method to many”

In this article I show how to describe DAD Essentials. Essentializing DAD is different from similar endeavors because Disciplined Agile (DA) is not a prescriptive method/framework like Scrum/SAFe. Instead, DA is a toolkit based on similar goals as Essence in that it is generative – both provide building blocks from which you can tailor and evolve a process that meets your needs.

The role of this Essentials is to provide an overview of the DAD guidance and over the capabilities that could be developed for each significant aspect (“Alphas”) of an Agile & Lean process. What is really happening on Agile/Lean adoption (and in any process improvement) is developing & exercising of new capabilities. What is specific to Disciplined Agile is that it provides guidance for developing the capabilities that team & organizations needs for their specific context.

DAD Essentials are presented here in cards format using the OMG/SEMAT Essence Language constructs as Alphas, Patterns, Resources, Activities, Work Products (see Glossary section below). Each “card” has a name, a description, and a list of capabilities for your team or organization to develop.

The following terms are used in Essence.

	Alpha	An essential element that is relevant to an assessment of the progress and health of the software engineering endeavor.
	Patterns	A generic mechanism / complex concepts that are made up of several other elements
	Resources	A source of information or content.
	Activities	One or more approaches for carrying out some work to be performed and can recommend actions on alphas and/or work products in order to and relevance this work.
	Work Products	An artifact of value and relevance for a software engineering endeavor.

Full delivery life-cycles

Full, beginning-to-end, delivery life-cycle it is an explicit representation of how a software release will progress in time. The pragmatism and effectiveness of Agile (<> Waterfall) are based on realistic progress milestones with the evolution of working software toward a consumable solution.  A team could have a reference lifecycle, but in a different context, they may need to use also other types of life-cycles.

Capabilities to develop:

• A reference life-cycle
• Support for Iterative-Agile, Lean, Continuous Delivery life-cycles
• Support for high incertitude deliveries
• Support for long term roadmaps (product, business, technology)

Consumable Solution

Consumable solution is more than working software. Consumable means that we meet stakeholder needs in the context constraints and it is usable, desirable, and functional. A solution implies that, as needed, we:  

• Develop high-quality software
• Provide new or upgraded hardware/platform
• Change the business/operational processes which the stakeholders follow
• Change the organizational structure in which our stakeholders work
• Update supporting documentation

Capabilities to develop:

• Understand Consumable Solution
• Build using DAD guidance the Consumable Solution across life-cycle milestones, considering various life-cycle and practices options depending on context

Adapt to Context

DA supports two principles that motivate you to adapt your approach to your context:

• Context counts. People, teams, and organizations are all unique – leads us to a critical idea that your process and organization structure must be tailored for the situation that you currently face.
• Choice is good. Different contexts require different strategies – teams need to be able to own their own process and to experiment to discover what works in practice for them given the situation that they face. This is why the DA framework presents people with choices through the application of process goal diagrams.

Capabilities to develop:

• Understanding Agile Scaling/Tailoring Factors
• Tailor WoW in context using DA Guidance
  • Select life-cycles
  • Select practices per process goals in a given context and life-cycle
  • Use DAD Library of practices & context options guidance
  • Make experiments to see what works in your context

Core Agile Practices

Core Agile Practices will help you have a Lean process: they address the main sources of waste and have multiple benefits at the same time. It is not a coincidence that XP is based on some of them, Disciplined Agile and Agile Modeling refer them as critical practices. (See also references from Mary Poppendieck / Tom Poppendieck in “Lean Software Development”). Some core practices are:

• XP practices: Small Releases, Pair Programming, Simple Design, Refactoring, TDD, Coding Standards, and more.
• DA/Agile Modeling practices: Requirements/Architecture Envisioning, Architecture Handbook, Model Storming, Rolling Wave Planning, and many more.
• Method free practices: Clean Code/Architecture.

Interestingly, Essence describes in detail several dozen core agile practices in detail whereas Disciplined Agile puts several hundred agile, lean, and even traditional core practices into context. This is one of several reasons why Essence and DA are complementary to one another.

Capabilities to develop:

• Understand how and why you will eliminate waste
• Context usage: Core Agile Practices are not “best practices” and we still need to know the trade-offs and options for different context (See Adapt to Context alpha). Make experiments to see what works in your context

Teal Teams and Organizations

Optimize the whole: the Organization (and its constituents teams) represents that “whole” where the work optimization really makes sense. In Reinventing Organizations, Frederick Laloux presents the historical evolution of organizations from tribal to modern agile approaches:

• Red (Magic/Tribal): Impulsive, survival urgency
• Amber (Traditional/Agrarian): Authoritarian, formal hierarchy
• Orange (Scientific/Industrial): Task-oriented, profit/grow focus
• Green (Post-Modern/Information): Value-based, consensus/participative style
• Teal (Self-Organizing/Adaptive): Cellular organism with evolutionary purpose

Disciplined Agile use this model and propose as a goal the Teal organization (or at least Green): cellular, self-organizing, adaptive, aware, with evolutionary purposes. Most likely, your organization is a “rainbow” (e.g. Orange/Green/Teal). Context counts, different teams faced different situations and you can choose your strategy. You want to be at least Green because that will provide – through a participative & collaborative style – a solid foundation for further process improvement. Also, the DA Principle, Be Awesome has some expectations:

• Treat people with respect, honesty, be reliable and open
• Willingly collaborate with others

Capabilities to develop:

• Teams will offer psychological safety with clarity about roles and responsibilities
• Cross-functional skills teams, where T-skilled “generalizing specialist” is the most pragmatic, effective, efficient approach.
• Use collaborative work to envision, look ahead and just-in-time clarifications.
• Collaborative and continuous process tailoring and improvement, not only on retrospectives
• Inter-teams’ collaboration: Communities of Practices and Centers of Excellence, etc.
• Individual, team, enterprise and community level awareness
• Use pragmatic agile roles
• Address team and organization level scaling factors

Guided Process Improvement

The purpose of the Continuous Improvement process blade is to enable people within your organization to easily share their improvement learnings with one another in a systematic way. The technique of Guided Continuous Improvement (GCI) shows teams how to leverage the DA toolkit to speed up their process improvement efforts.

Capabilities to develop:

• Continuous improvement must be explicit and fundamental
• A base support for improvement should be running: life-cycles, collaborative work, retrospectives
• Kaizen strategy: continuous improvement should always be running in small steps and experiments. This lean strategy is fundamental for addressing problems complexity & incertitude.
• Guided Continuous Improvement (GCI)
  • apply the DAD toolkit to adapt to context (see corresponding alpha) for process goals as: selecting a life-cycle, forming the team, addressing changing stakeholders needs.
  • hire/listen experienced coaches
  • make progress on adopting Core Agile Practices
  • have explicit goal/guidance for Evolving your WoW

An Agile method such as Scrum, or a framework such as SAFe, can be a good start but it will have too few guidelines for choosing your way of working (WoW) in context or too little guidance for core Agile practices. DA provides the guidance for evolving your WoW and Essence the details regarding core practices. As Ivar Jacobson likes to say, this will enable you to break out of “method prison.”

Patterns

Resources

Work Products

Activities

I want to thank Scott Ambler who started this Essentializing DA initiative and collaborated with SEMAT from 2009. Scott helped me with feedback and review of current materials. DA Essentialization began with the example of the DB Refactoring technique earlier this year.

• [R1-ES] Essence Specification 1.2 – https://www.omg.org/spec/Essence/1.2/
• [R2-SM] SEMAT (Software Engineering Method and Theory) – http://semat.org/
• [R3-SMMS] SEMAT mission statement: http://semat.org/mission-statement
• [R4-ESSDB] Essentialize DB refactoring –  Database Refactoring
• [R5] DA approach for organizations – Teal is the New Black

Copyright statement

Use of Essence – Kernel and Language for Software Engineering Methods Specifications is the subject of Term and conditions & Notices found at https://www.omg.org/spec/Essence/1.2

What is the point of the retrospective?

The retrospective is one of the most important ceremonies in all of agile.  This is the time the team spends together to assess how they are working together and define steps to improve that process.  It needs to be a “safe place” where people are able to speak openly and honestly.  This is their opportunity to air their dirty laundry and work through their inter-personal issues.  This is a time of growth for the team and for the team to take ownership of improvement.  The team lead will facilitating the retrospective and should manage the interactions to keep the environment safe.

Define the Team

If the retrospective is a team ceremony, then what do we mean by team?   The team includes: the team lead, the architecture owner and all other members that actively contributed to meeting the deliverables for the iteration.  This includes: developers, testers, BAs, QAs, or other specialists such as technical writers, database engineers etc.

What about the Product Owner (PO)?

The PO is NOT a member of the team.  They certainly interact with the team but they do not contribute to meeting the deliverables for the iteration so they are not a member of the team.  They are not allowed to participate in the planning poker for the user stories for the same reason.  The team votes because they are on the hook for delivering based on the sizing and the estimates but the PO is not on the hook, so they don’t get a vote.

Should the Product Owner participate in the retrospective?

In general, I would start by including the PO in the retrospectives because the team does have to learn and adjust to working with the PO. Keep in mind though that the PO may come and go but the team should stay together so it is most important that the team works well together.??As a coach, I usually talk to the PO beforehand to say that they are an invited guest and that it is a privilege to be part of the meeting so they should act accordingly.   I have been in many situations where the PO was welcomed at the retrospective, and felt left out if not included.  I favor building trust between the business (the PO is their representative) and IT (the team).  Including the PO in the retrospective can help the PO assimilate with the team.

I have also had several situations where as the coach I had to ban the PO from the retrospective because they were too commanding and disruptive in the meeting for the team to have an effective retrospective.  I have also seen many situations where the PO is also the resource manager of members of the team (which in itself is not recommended).  Having managers in the room can definitely have a dampening effect on the member’s willingness to be open and honest about problems and solutions.

If the PO doesn’t participate, at least as an observer, the team runs the risk of having to “sell” the cost of their improvement actions (against other backlog items) after coming up with them. Hopefully the PO is engaged enough with the team to understand its weaknesses and support improvement in those areas whether then attend the meetings or not.

Team Decision

Retrospectives are about improving the process, and a non-trivial part of that is optimization of collaboration between the PO and the team.  I would suggest that the team should decide whether or not to include the Product owner.

What about the Stakeholders?

The retrospective should absolutely be a closed door session for the stakeholders since the retrospective must be a “safe space”.

There was a twitter debate lately that talked about a team being subjected to “a drive by criticism from 2 PM’s during a Retrospective”.  This is a good reminder why we constrain attendance.  The “safe place” is affected by the presence of people with positional authority, potential agendas or other implicit impact.??The team may decide to invite such people – usually to ensure that they are communicating improvements needed that are beyond their locus of control.??Having outsiders as guests at the retrospective will change the dynamics but at least it is a team decision to do so.

It is very important that the team own their process.  If they’re uncomfortable that someone is in the room then that person should be asked to either change their behavior or leave (perhaps to be invited back in the future).??The coach should always be thinking along the lines of “do we have the right people in the room” and then act accordingly

Isn’t agile all about transparency?

There was a twitter debate lately that centered on transparency.  I believe that transparency is a key element to making agile successful.   I’m all for transparency in everything about agile; EXCEPT the retrospective!??Sometimes you need to have a family meeting outside of the public eye and that is the retrospective. The retrospective is all about resolving your issues in private so that you can present a united front to the rest of the world. To use a sports analogy, an NHL coach doesn’t invite the business (fans) into the dressing room between periods.  There are lots of other places for transparency; the retrospective doesn’t have to be one of them.

The output of the Retrospective

While the actual retrospective meeting is closed to other observers, I would suggest that the action items coming out of the retrospective need to be made public and posted as an information radiator for everyone to see.  The changes are more likely to get implemented if the team sees them every day.  The team may also want to “radiate” their improvement actions on their dashboard.

The actions and results of the actions may also be shared with other teams through what is often called a Retrospective of Retrospectives. I encourage teams to only choose one or two areas for improvement at a time to provide focus and make meaningful progress.

As mentioned in a previous post on Team Member Rights, transforming to Agile is a culture change, and all cultures have rules so that everyone understands their expected behavior. DAD has inherited some of the basic rules from the XP world. Each time I present these rules in the training course, inevitably they end up on the list of things that “freaked us out”. That means to me that the rules are really important and that the community needs to be talking about them and getting a clear understanding and acceptance.

The Responsibilities of Everyone:

1. To produce a solution that meets the needs of stakeholders given the resource constraints. The primary goal of the team is to meet the stakeholder needs as best they can.
2. To optimize the use of those resources. Optimizing the resources should be looked at from a long term perspective. Tasking the specialist on the team with a particular type of task may seem optimal in the short term because the task is done quickly. In the long term it may be more optimal to assign a new team member to train with the specialist to do that type of task even if it takes longer to complete the current task.
3. To be willing to collaborate extensively within your team including those outside your specialty. Gone are the days when a developer could go hide in a corner and hack away at code. Collaboration is required across all team members.
4. To share all information including “work in progress”. One of the keys to successful Agile is transparency. If you run into a problem, tell the team so they can help out or at least re-plan. If you complete something quickly, tell the team so they can work on next steps whether that be: testing, or documenting or promoting. Everyone should know what everyone on the team is doing all the time.
5. To coach others in your skills and experience. The goal is to increase the skill set of everyone on the team so be prepared to teach your skills to the other team members. Previously the performance of people was assessed on their ability to perform specialty tasks. The focus needs to shift to how well a person collaborates, shares and increases the teams ability to perform.
6. To expand your knowledge and skills outside your specialty. Again, the goal is to increase the skill set of everyone including you so everyone needs to be open to learning new skills so they can help the team do every required task.
7. To validate your work as early as possible, working with others to do so. No more writing code and getting it promoted directly to production or producing documents that haven’t been reviewed. All code should be reviewed by another developer; non-solo development is great for this because it reduces the feedback loop to almost zero since 2 sets of eyes are always on the code. And all code needs to be tested against the acceptance criteria in the user story. Nothing goes out without someone else on the team reviewing it because it is a team responsibility to ensure quality.
8. To attend co-ordination meetings in person or through other means if not collocated. The co-ordination meeting is the most important meeting of the day and nothing else takes priority. Get to the meeting and get there on time. If you are late then you are holding up the whole team.
9. To proactively look for ways to improve team performance. The retrospectives are designed to fix and improve the team’s process. Come prepared to the meeting to discuss how the iteration went and talk about things that went wrong and how to fix them. However, if you see something during the iteration that can be fixed, don’t wait for the retrospective!! Bring it up at the co-ordination meeting and suggest a fix right away.
10. To avoid accepting work outside of the current iteration without consent from the team. The team commits to complete a specific bundle of work when they start an iteration. That means that all team members have made a commitment to complete all the work in the iteration. If you as a team member take on work from outside the iteration (whether it be from a colleague or an old boss or your current boss) then it means you are not working on tasks for the iteration and you are letting your team down. You should refuse all outside requests for your time. If that doesn’t work tell them they need to talk to the team lead. The team lead should refuse the request. If that doesn’t work then tell them to talk to the Product Owner. The Product Owner should refuse the request but offer to write it up as a user story and put it onto the backlog for consideration.

Transforming to Agile is a culture change and all cultures have rules so that everyone understands their expected behavior. Disciplined Agile (DA) has inherited some of the basic rules from the XP world. Each time I present these rules in the training course, inevitably they end up on the list of things that “freaked us out”. That means to me that the rules are really important and that the community needs to be talking about them and getting a clear understanding and acceptance.

Let me first address The Rights of Everyone:

1. To be treated with respect. Whether you are going Agile or not, this is a good thing to promote. Let people be heard, don’t talk over people, listen to what they have to say, no name calling etc etc. Every one of your team mates brings something of value to the team so show them the respect you expect in return.
2. To work in a “safe environment”. This goes hand in hand with treating people with respect. Everyone needs to feel they are in a safe environment, that they can offer input without being mocked or ridiculed. For everyone to freely collaborate they need to know that their input is valued.
3. To produce and receive quality work based upon agreed standards. Establish the work standards up front. Make sure everyone on the team understands the standards and agrees to the standards. No one can be expected to meet moving standards.
4. To own the estimation process. The people doing the work get to do the estimations so that estimations are not being imposed on the team
5. To determine how teams resources are allocated. The team has the right to determine who is going to work on what task when. The team knows the strengths and weaknesses of the team better than anyone else so they know best, how to meet the team goals
6. To be provided good faith information and decisions in timely manner. For the team to be effective they need good information to work with and information must be available when they need it. Any delays in providing requirements or answers to questions will reduce team productivity.
7. To own the team’s processes and be enabled to improve them. DA recommends an initial process but as the team matures the team has the right to adjust and improve the process so that it works for them. The whole point of doing retrospectives is to have the team reflect on how well the last iteration was performed, keep doing the activities that worked and fix the activities that didn’t.

See the post Responsibilities of Everyone where I discuss the other side of the coin and look at responsibilities that pay for these rights.

In this continuing series about how the Disciplined Agile (DA) tool kit supports and enhances DevOps, we overview several strategies to help your organization adopt a Disciplined DevOps strategy:

1. Invest in your people.  In our experience 80 to 90% of your overall effort will be in helping people to learn new skills and ways of thinking and to rethink if not abandon many of the “best practices” of yesteryear.  This requires training, education, and coaching over a long period of time – most people will require many months, and sometimes years, to make the transition to this new mindset.
2. Create a safe learning environment.  Teams must be free to experiment, to try new strategies to discover what works for them in the situation that they face.  Very often this will work out well, with a few stumbles along the way, but every so often the experiment will show that the strategy just isn’t right for this team.  That should still be considered a successful experiment, learning what doesn’t work is just as valuable as learning what does, and the team should not be worried about recrimination for “failed” experiments.
3. Look at the “whole system”.  Disciplined DevOps is about more than just continuous delivery (although that is a great start) and in most enterprises it’s about more than just streamlining development and operations.  With a Disciplined DevOps mindset we strive to improve flow through the entire ecosystem, including development, operations, support, enterprise architecture, data management, release management, and most importantly to the business itself.
4. Improve locally, transform globally.  Each team, including your solution delivery teams, your enterprise architecture team, your operations staff, and many others must strive to improve and streamline the way that they work.  These local improvement efforts must be supported by a “global” transformation effort that focuses on improving DevOps across your entire ecosystem.  Every team will affect other teams, motivating them to make improvements which in turn affects how they work with others.  Your IT department is a complex adaptive system where people and teams learn and improve over time in a dynamic, evolutionary manner.  If you just focus on local improvements your DevOps effort is likely to devolve into chaos.  If you just focus on a company-wide, global transformation it is likely to get bogged down in bureaucracy.  An “improve locally, transform globally” approach is a viable middle ground that benefits from these two extremes while avoiding the disadvantages.
5. Have a communication plan (and work it).  Adopting a Disciplined DevOps strategy within your organization typically requires a lot of (often small) changes.  Although it may be clear to you why this is important it isn’t always clear to everyone else.  People need to understand why you’re making these changes, what’s in it for them, what the overall change strategy is, how far along the plan you currently are, what changes are coming soon, and so on.  You communication plan may include regular newsletters, posters overviewing key concepts, brown bag lunches where people share their experiences, electronic discussion forums, management presentations, and many more.  The keys to success are to have a constant drum beat of information, to be as open and honest about what is actually occurring, to provide opportunities to everyone to learn, and to motivate everyone to share their learnings.
6. Think long-term.  Disciplined DevOps is a journey, not a destination.  It takes time for people to adopt a new mindset, months and often years before it is truly ingrained in their way of thinking.  This paradigm shift does not occur by management fiat, nor does it occur as the result of a day or two of training (although training is important), nor does it occur because you’ve invested in new tooling.  Organizations that successfully make this paradigm shift do so by investing in their people, their process, and their tooling over the long term.