Disciplined Agile

In our previous blog, Continuous Improvement Through Experimentation, we worked through how teams can evolve their way of working (WoW) through experimentation and kaizen.  Figure 1 depicts the logic of a single pass through a Plan-Do-Study-Act cycle for doing so. The team identifies a potential way to improve their WoW, they experiment with it for a bit, they assess how well it worked for them, then they keep what works well and drop what doesn’t.

Figure 1. Continuous improvement through experimentation (click to enlarge).

Figure 2 shows how the effectiveness of a team’s WoW rises over time via this experimentation-based strategy.  When an experiment with a new technique works (the experiment is successful) the team’s effectiveness increases.  When an experiment “fails” their effectiveness dips for a bit but then rises back to where it was once they go back to their previous WoW.

Figure 2. Team effectiveness improves over time by experimenting with new WoW (click to enlarge).

So what can we do to improve on this? The linchpin is the very first step in the process of Figure 1, the identification of a technique to experiment with.  As you see in Figure 3, when we improve the likelihood that a technique will work in our situation then our effectiveness rises faster due to more successful experiments.  We call this guided continuous improvement.

Figure 3. Guided continuous improvement increases the chance of successful experiments (click to enlarge).

It’s a simple idea – with better process decisions we achieve better process outcomes, as you can see in Figure 4. There are three ways that you can do this:

1. Hire an experienced coach (and listen to them). Although it can be
   hard to find an experienced agile coach they do exist and if you’re lucky enough to have one then follow their guidance.
2. Apply the Disciplined Agile (DA) toolkit. There are several ways you can apply the DA toolkit to help you to make better process decisions.  Instead of prescribing the “best practices” you must adopt, DA instead advises you on process-related issues you need to think about and the options available to you.  Such issues include what you should consider when choosing a lifecycle, what you should consider when forming your team, and what you should consider when addressing changing stakeholder needs (to name a few important things). The DA toolkit then presents you with potential options and the trade-offs associated with those options.  This gives you an idea as to what techniques you might want to experiment with and what is likely to happen if you do, enabling you to make better process decisions. This site overviews these decisions, as you can see by following the previous links, and the book Choose Your WoW! summarizes the trade-offs for Disciplined Agile Delivery (DAD).
3. Both of the above. Good coaches have the humility to recognize that they don’t know everything, and will leverage the DA toolkit to help your team to make better decisions about new ways of working (WoW) to experiment with.

Figure 4. Team effectiveness improves at a quicker rate with guided continuous improvement (click to enlarge). 

Continuous improvement, evolving your WoW through experiments, is a proven way to achieve lasting process improvement. Lean practitioners have been doing this for decades and virtually every DevOps case study advises you to evolve your WoW this way. Guided continuous improvement takes it one step further and streamlines your experimentation efforts.

MORE INFORMATION

For more information about choosing and evolving your WoW, we humbly suggest that you consider reading our book Choose Your WoW! A Disciplined Agile Delivery Handbook for Optimizing Your Way of Working. If you want to succeed at enterprise agile you need choices, not prescriptions.

A fundamental philosophy of agile is that teams should own their own process, or as we like to say in Disciplined Agile (DA) teams should choose their way of working (WoW).  Of course this is easier said than done in practice. The challenge is that every team is unique and faces a unique situation – in other words, context counts. Furthermore, there are no “best practices,” rather, every practice has tradeoffs and works well in some situations and poorly in others. Worse yet, you really don’t know how well a technique will work for you until you actually try it out in your environment.  Given all of this, how can a team choose its WoW?

Since the 1980s the lean community has shown that an effective way to evolve your process is to do so as a series of small incremental improvements, a strategy called kaizen.  Numerous organizations have taken this approach over the years, and virtually every single DevOps success story is based on a multi-year kaizen-based continuous improvement strategy. Figure 1 depicts the workflow of implementing a single improvement, with Deming’s Plan-Do-Study-Act (PDSA) loop on the right-hand side to indicate the iterative nature of the overall process.

Figure 1. Running an experiment to evolve your WoW (click to enlarge).

Let’s explore each step one at a time:

1. Plan: Identify a potential improvement. The team identifies a technique, either a practice or strategy, that they believe will work for them. This may be something someone on the team has done before, something they’ve read about, or even an idea that they’ve identified on their own.  An important consideration is to try a technique where it is “safe to fail” at it – you shouldn’t be putting your team at risk by experimenting with a new WoW.  If an experiment is deemed to risky to try, attempt to  break it down into a series of smaller experiments that are less risky. It is also critical to identify the criteria against which you’re going to assess the effectiveness of the technique.  Ideally most of the criteria is quantifiable in nature, although don’t ignore qualitative measures too.
2. Do: Try out the new WoW (experiment). The team needs to see how well the technique works for them in their environment.  Do they have the skills to perform the technique? How well does it work given their technology platform?  How well does it work given their organization and team culture? The team needs to give the experiment sufficient time to determine how well it works in practice. This could several anywhere from several days to several months, although in most cases several weeks is sufficient.
3. Study: Assess effectiveness. After you’ve run the experiment you should assess how well it worked for you. This assessment should be against the criteria that you agreed to at the beginning.
4. Act: Adopt or abandon the new WoW.  Sometimes you’ll find that a new technique works incredibly well for you, and other times you’ll experience the other extreme and discover that a technique is an abysmal failure for you.  But in most cases there will be some aspects of the technique that work well for you and other aspects that do not (an indication that maybe you need to consider running a new experiment to identify a solution). Our advice is to adopt what works well for you and to abandon or better yet improve upon what doesn’t.
5. Act: Share learnings with others. DA teams are enterprise aware, recognizing that they are only one of many teams within your organization. So, when a team learns something about a technique the implication is that they should share it with others.  Strategies for doing so are addressed by the Evolve WoW process goal.

Figure 2 shows how your team’s effectiveness improves over time with a continuous improvement approach.  When you experiment with a new technique and it works out well for your team then your team effectiveness improves.  When an experiment “fails” your team effectiveness dips for a bit – the technique didn’t work well in your situation – but then you end the experiment and go back to your previous way of working (your team’s level of effectiveness goes back to where it was).

Figure 2. Experiment to evolve your WoW (click to enlarge).

When you keep at it, when you adopt a kaizen mindset, your team effectiveness increases over time as we see in Figure 3. The figure also shows that when you first adopt a continuous improvement strategy that your team effectiveness drops at first because you’re learning how to follow the continuous improvement process of Figure 1.  In many ways you begin this improvement journey by experimenting with this experimentation-based strategy.

Figure 3. Continuous improvement over time (click to enlarge).

Some organizations struggle with the idea of experimentation, likely because they still believe in the idea of “best practices” and often because they’re looking for an easy answer. They’re afraid to experiment because they might “fail,” not realizing that a failed experiment teaches you what doesn’t work for your team given your current situation. Running small, “safe to fail” experiments is absolutely critical for improving your WoW.

Where this blog overviewed the strategy of Continuous Improvement, in the next blog in this series we will see how better decisions lead to better outcomes via Guided Continuous Improvement.  Stay tuned!

More Information

For more information about choosing and evolving your WoW, we humbly suggest that you consider reading our book Choose Your WoW! A Disciplined Agile Delivery Handbook for Optimizing Your Way of Working. If you want to succeed at enterprise agile you need choices, not prescriptions.

The Disciplined Agile (DA) tool kit references hundreds of agile, lean, and in some cases “traditional” techniques, putting them into context. One such technique is database refactoring, adopted from the Agile Data (AD) method, which is critical for the Improve Quality process goal. Where a code refactoring is a simple change that improves the quality of code, a database refactoring is a simple change to your database that improves the quality of its design without changing the semantics in a practical manner. Database refactoring is a sophisticated practice, in fact I co-wrote a 400+ page book about it, Refactoring Databases: Evolutionary Database Design, with Pramod Sadalage in 2006. The book, which won a Jolt Productivity Award, provides a detailed description of the practice. But what if you’d like to start with something simpler before jumping into the details? This is where Essence comes in.

A few months ago a group of us – Roly Stimson of IJI, Nebojsa Trninic, Vladimir Savic, Ervin Varga, and myself – decided to experiment with the idea of essentializing DA. To focus our work we decided to start with a simpler problem, that of essentializing a subset of the techniques referenced by DA, in this case those of Agile Data. We then focused further on a minimal viable product (MVP), in this case the single practice of database refactoring.

We captured the practice using Essence Enterprise 365 from IJI, a screen shot from which is shown below. Essence is a language for describing software engineering methods and practices from Ivar Jacobson International (IJI). Essence was created as a part of Software Engineering Method and Theory (SEMAT) and approved by The Object Management Group as a standard in 2014. I’ve been working with SEMAT from its beginnings in 2009 and have known (and admired) Ivar since the late 1990s.

As you see in the screen shot we summarized database refactoring using the tool, describing the practice as a collection of cards. The tool can be used to develop process material, as we’ve done, and more importantly it can be used by teams to tailor and evolve their own process. Within the tool are dozens of practices, including ones from Scrum and XP, that a team can configure to reflect their own way of working (WoW).

In the following picture you see that these cards can also be printed out and used in a physical manner, often for process tailoring or training. In November I experimented using the cards to train a small group of people. We played a game where the group worked through how they would go about refactoring an existing database. They discussed each card, moving them around on the table until they had something that looked like the picture below. The cards provided a tactile way for them to explore database refactoring by thinking through how they would apply it in practice. IJI has identified a collection of teaching and process tailoring games using cards such as this.

An interesting result of this teaching experiment is that we discovered that we wanted more cards. The group wanted cards to explore automated database regression testing, agile data modelling, and continuous database integration in detail, and of course how all of these things fit together. So it looks like we have our work cut out for us.

We’d love to hear what you think about this effort. This blog has been a brief description of our work to essentialize the practice of database refactoring, but we have a lot more work ahead of us. A more detailed description will be presented in a forthcoming research paper that the team is working on. We also intend to continue with this essentialization effort to share some of the key DA techniques within Essence. Stay tuned!

Our new book, Choose Your WoW! A Disciplined Agile Delivery Handbook for Optimizing Your Way of Working, is now available!

This handbook is an indispensable guide for agile coaches and practitioners to identify what techniques – including practices, strategies, and lifecycles – are effective in certain situations and not as effective in others. This advice is based on proven experience from hundreds of organizations facing similar situations to yours.

There are literally hundreds of books that have been written on agile and lean. So why one more? Most books focus on just a subset of what is required to delivery end-to-end solutions in an agile manner. Developing a coherent approach to the complete picture by piecing together these practices is difficult, especially when the advice in many of these books is conflicting and not well researched, often based on just a few successful applications in a specific context.

Disciplined Agile (DA) is the only toolkit available that combines the most effective practices across lean and agile methods into one comprehensive context-driven approach that you can use to optimize your way of working (WoW).

This book is organized into several sections:

1. Disciplined Agile in a Nutshell. This section describes guided continuous improvement, the Disciplined Agile (DA) Mindset, and of course DA itself.
2. Successfully Initiating Your Team. This section describes strategies for how a team can do just enough work to get themselves organized and to come to a general agreement around the scope, architectural strategy, and plan for the current release.
3. Producing Business Value. This section describes a myriad of agile and lean strategies for producing a minimal marketable release (MMR) of a consumable solution that is ready to be transitioned into production or the marketplace.
4. Releasing Into Production. This section describes techniques to successfully release a consumable solution into production or the marketplace. Ideally this is a fully automated activity that runs in minutes or hours.
5. Sustaining and Enhancing Your Team. This section describes strategies that support the team and/or help to make it more effective. This includes techniques for helping to grow individual team members, for evolving your team’s WoW, and for coordinating both within the team and with other teams in your organization.

For more information, please visit the Choose Your WoW! book page.

On Monday I attended a talk by Mike Rosen about successful Digital Transformation.  As always, Mike’s presentation was very insightful.  One of many points that he made is that most organizations are struggling with the current talent shortage, something that in my experience has been a problem worldwide for several years and will continue to be so.

There are many reasons for why there is a talent shortage:

1. It takes years to master your craft.  In the distant past, the 1980s, I had to go to university for four years to become a junior programmer.  Then I had to work for years to gain the experience and skills to become a developer. And things are certainly more complex now than it was back then. My point is that it takes years, not days or weeks, to gain the skills and experience that organizations are looking for.
2. Organization’s planning horizons are too short. Demand has outstripped supply for many years within IT, and we’re clearly seeing this within the agile space. Having said that, the growth of IT has followed a steady and predictable curve for years that organizations could have planned for, and more importantly executed on, had they chosen to. But, the multi-year time frame to grow talent doesn’t mesh well with the quarterly-results focus of most organizations.
3. Organizations have cut back on training. In the 1980s and most of the 1990s it was common for IT people to be given two or more weeks of training every year.  This was slowly whittled away down now many people are lucky to be given the funding to attend a single workshop or one-day conference each year.  Yes, there are some great resources online that can provide a substitute for training, so that’s bit of help. Unfortunately most organizations expect people to learn on their own time.  Some people do in fact do that, but many do not. I regularly meet people at conferences who have taken a vacation day and spent their own money to attend the event – it’s laudable that they’re doing this, but incredibly frustrating that they’ve had to resort to doing so. My point is that if your organization has not been investing in its people it seems disingenuous for it to complain about the lack of skilled people available to it.
4. Apprenticing is virtually non-existent in IT. Many other professions have a culture of apprenticing to help bring new people in and help them to gain the skills and experience required. Although I have seen several organizations attempt to institute a mentorship program, and have seen limited successes at doing so, I have never run into an organization with active apprenticing. Non-solo work practices such as pair programming, mob programming, and modeling with others are arguably the start at apprenticing practices within the agile space so perhaps this is how it begins in IT.  Time will tell.

Here are a few things that your organization can do to start addressing its talent shortage:

1. Become an attractive place to work. It’s important to note that attractive organizations, the ones that are known for doing interesting work and for providing their people with autonomy to learn and to choose their way of working (WoW), don’t seem to have problems attracting and retaining good people.  Granted, this doesn’t grow the pool of talented people but it does help to address your challenge.
2. Start investing in your people long term. If you want talented people in your organization you need to grow and nurture them yourself.  You can do so by providing coaching, mentoring, training, and education opportunities.  You can support the creation of communities of practice (CoPs), also called guilds, and create centres of excellence (CoEs).  Motivating teams to experiment with non-solo work practices such as pairing and mobbing will enable them to learn from each other.  And of course all of these strategies require investment of time and money.
3. Become a learning organization. If you want people to grow their skills you have to allow them to do so.  Allow them to experiment with new ways of working (WoW), recognizing that some of those experiments will “fail” and you’ll discover what doesn’t work for you in your context. Also recognize that individuals and teams have to be allowed to choose their WoW, and to evolve it as they learn, so that they can improve.

The primary reason why there is a shortage of talented people is because organizations are underinvesting in the learning paths of their staff.  If you want talented people you’re going to have to help create them.

Related reading:

• The People Management process blade
• The Grow Team Members process goal