Disciplined Agile

The Disciplined Agile (DA) tool kit references hundreds of agile, lean, and in some cases “traditional” techniques, putting them into context. One such technique is database refactoring, adopted from the Agile Data (AD) method, which is critical for the Improve Quality process goal. Where a code refactoring is a simple change that improves the quality of code, a database refactoring is a simple change to your database that improves the quality of its design without changing the semantics in a practical manner. Database refactoring is a sophisticated practice, in fact I co-wrote a 400+ page book about it, Refactoring Databases: Evolutionary Database Design, with Pramod Sadalage in 2006. The book, which won a Jolt Productivity Award, provides a detailed description of the practice. But what if you’d like to start with something simpler before jumping into the details? This is where Essence comes in.

A few months ago a group of us – Roly Stimson of IJI, Nebojsa Trninic, Vladimir Savic, Ervin Varga, and myself – decided to experiment with the idea of essentializing DA. To focus our work we decided to start with a simpler problem, that of essentializing a subset of the techniques referenced by DA, in this case those of Agile Data. We then focused further on a minimal viable product (MVP), in this case the single practice of database refactoring.

We captured the practice using Essence Enterprise 365 from IJI, a screen shot from which is shown below. Essence is a language for describing software engineering methods and practices from Ivar Jacobson International (IJI). Essence was created as a part of Software Engineering Method and Theory (SEMAT) and approved by The Object Management Group as a standard in 2014. I’ve been working with SEMAT from its beginnings in 2009 and have known (and admired) Ivar since the late 1990s.

As you see in the screen shot we summarized database refactoring using the tool, describing the practice as a collection of cards. The tool can be used to develop process material, as we’ve done, and more importantly it can be used by teams to tailor and evolve their own process. Within the tool are dozens of practices, including ones from Scrum and XP, that a team can configure to reflect their own way of working (WoW).

In the following picture you see that these cards can also be printed out and used in a physical manner, often for process tailoring or training. In November I experimented using the cards to train a small group of people. We played a game where the group worked through how they would go about refactoring an existing database. They discussed each card, moving them around on the table until they had something that looked like the picture below. The cards provided a tactile way for them to explore database refactoring by thinking through how they would apply it in practice. IJI has identified a collection of teaching and process tailoring games using cards such as this.

An interesting result of this teaching experiment is that we discovered that we wanted more cards. The group wanted cards to explore automated database regression testing, agile data modelling, and continuous database integration in detail, and of course how all of these things fit together. So it looks like we have our work cut out for us.

We’d love to hear what you think about this effort. This blog has been a brief description of our work to essentialize the practice of database refactoring, but we have a lot more work ahead of us. A more detailed description will be presented in a forthcoming research paper that the team is working on. We also intend to continue with this essentialization effort to share some of the key DA techniques within Essence. Stay tuned!

Our new book, Choose Your WoW! A Disciplined Agile Delivery Handbook for Optimizing Your Way of Working, is now available!

This handbook is an indispensable guide for agile coaches and practitioners to identify what techniques – including practices, strategies, and lifecycles – are effective in certain situations and not as effective in others. This advice is based on proven experience from hundreds of organizations facing similar situations to yours.

There are literally hundreds of books that have been written on agile and lean. So why one more? Most books focus on just a subset of what is required to delivery end-to-end solutions in an agile manner. Developing a coherent approach to the complete picture by piecing together these practices is difficult, especially when the advice in many of these books is conflicting and not well researched, often based on just a few successful applications in a specific context.

Disciplined Agile (DA) is the only toolkit available that combines the most effective practices across lean and agile methods into one comprehensive context-driven approach that you can use to optimize your way of working (WoW).

This book is organized into several sections:

1. Disciplined Agile in a Nutshell. This section describes guided continuous improvement, the Disciplined Agile (DA) Mindset, and of course DA itself.
2. Successfully Initiating Your Team. This section describes strategies for how a team can do just enough work to get themselves organized and to come to a general agreement around the scope, architectural strategy, and plan for the current release.
3. Producing Business Value. This section describes a myriad of agile and lean strategies for producing a minimal marketable release (MMR) of a consumable solution that is ready to be transitioned into production or the marketplace.
4. Releasing Into Production. This section describes techniques to successfully release a consumable solution into production or the marketplace. Ideally this is a fully automated activity that runs in minutes or hours.
5. Sustaining and Enhancing Your Team. This section describes strategies that support the team and/or help to make it more effective. This includes techniques for helping to grow individual team members, for evolving your team’s WoW, and for coordinating both within the team and with other teams in your organization.

For more information, please visit the Choose Your WoW! book page.

One of the key benefits of Disciplined Agile is its flexibility to apply different lifecycles, practices, and strategies based on the vast array of situations that a typical enterprise faces. It is “pragmatic agile” after all. Unfortunately many organizations box agile into a corner. Many try to standardize on one flavor such as Scrum/XP, Kanban, or worse, a prescriptive scaling pattern such as SAFe. We also sometimes see project authorities restrict agile to initiatives that pass all of a certain criteria such as; teams of less than nine people, full-time Product Owner, requirements not needed beyond user stories, collocated team, whole team, and other characteristics of “classic agile”. Using such restrictive selection criteria means that the percentage of teams that are permitted to adopt agile is often far less than if a more pragmatic approach to applying agile was permitted.   As a result we see situations whereby a large percentage of projects that could benefit from agile approaches are forced to use traditional methods. This marginalizes the adoption of agile in such enterprises as exceptions rather than the default approach that it should be.

We know that the majority of enterprises can benefit from each of DA’s lifecycles. There are some situations where a basic Agile/Scrum approach is the best choice, but in others where the work is difficult to plan a Lean lifecycle would be a better fit. On other teams that have good Disciplined DevOps technical practices in place the advantages of the Continuous Delivery:Lean lifecycle could be applied. Additionally, many organizations have leading edge initiatives such as mobile applications where the market demand of features is not known so it may be that the Exploratory lifecycle is the best fit.  Different teams require different approaches.  Choice is good.

Organizations that have adopted Disciplined Agile appreciate this flexibility and over time their mix of lifecycles changes as they move their the projects in their application portfolio from tradition to basic, and eventually to the advanced continuous lifecycles. The following diagram depicts an actual strategy of one of our customers, which shows their expected evolution to a mix of lifecycles as their adoption proceeds and their agile capabilities improve.

As you can see from this example this organization expects to use Agile/Scrum for the majority of new agile initiatives but gradually increase the mix of the more advanced Continuous Delivery lifecycle as the teams’ capabilities improve. This is a typical pattern that we see for companies that are adopting Disciplined Agile, and in fact our new book Introduction to Disciplined Agile Delivery describes a team’s process improvement journey doing exactly that. The flexibility to adopt different lifecycles and apply a pragmatic and measured approach to adopting agile practices means that a far greater of percentage of projects can benefit from agile.

In my experience in running dozens of process tailoring workshops, over several years, with of teams of every shape size and experience level and in different organizations, the most recurring comment is that the workshops “revealed all kinds of options we didn’t even realize were options!”   Although almost always a bit of a hard sell at the outset, I have yet to work with a team unable to quickly grasp and appreciate the value of these activities.

I had to do quite a bit of experimenting in order to get the timing and content of the workshops right – and learned over time that success is also predicated on knowing whom to include when. My first attempts were gruelling, close-to-full day affairs with entire teams in attendance, held at or close to project kickoff. Though transparent and inclusive, to my surprise this approach was actually deemed a waste of their time by many team members, especially those whose contribution would occur primarily in the construction phase. First lesson learned – A technical team lead, architect or senior developer can actually stand in for most of the developers in the early stages. I find it helpful to always bear in mind what George Dinwiddie (http://www.velocitypartners.net/blog/2014/02/11/the-3-amigos-in-agile-teams/) dubbed “the 3 amigos” in determining who needs to attend a process tailoring session. Be it at inception, construction, or even in transition, you need to tailor not only the processes, but also the attendance of the workshop in order to ensure you have the right mix of people, with the right collaborative mindset, to cover issues pertaining to 1) the business problem being addressed  2) the potential technical solutions to that business problem and 3) the processes (both team and organizational) that will enable the work to be carried out.

My second lesson learned pertained to the format and presentation of the process blades themselves. I found that simply working from the published process maps was insufficient, as we ran into onerous issues around how to best record the WoW choices teams were making. I eventually reproduced the entire process blade library in a spreadsheet format, with columns for comments. This seemingly innocuous administrative step quickly ushered in the third lesson learned – the sessions can be used not merely to document an immediate WoW decision, but also to identify future, more “mature” aspirational choices which the team can set as goals over a specified time period.

A fourth lesson learned, and one that was also enabled by using a simple spreadsheet tool, is that it became far easier to Align with Enterprise Direction. By “locking down” enterprise-level process choices across all the blades where applicable, a lot of potentially fruitless (at that point in time) discussion was saved for many a team. No use in discussing test automation strategies to death for instance in divisions still completely relying on manual tests, or at the opposite end of the spectrum, teams endowed with high-performing, well-integrated CI/CD environments. This is a large part of what DA calls self-organization within appropriate governance.

The fifth and final major lesson learned was to never start from a blank slate if at all possible. I would typically show up at a team’s first process tailoring workshop with a pre-filled version from another team facing somewhat similar challenges (the identifying data being scrubbed so they could not identify the previous team). I would then challenge the new team to reflect on the choices and determine whether they made sense for their own context. This also saved time and effort, as there are recurring themes and common challenges within organizations that all teams face.

Here’s an important note on determining participation – Ultimately, the teams themselves are the best arbiters of who should attend the sessions at varying stages of advancement. Allowing this will typically result in a bit of initial over participation, followed by under participation (especially is the pressure is on to get “real” work done!) – the key as facilitator is to coax the team back into balanced participation, and to lobby the organization for the necessary support in freeing people up. The support will become easier and easier to obtain as the benefits of allowing teams to choose their WoW become apparent.

Finally, be prepared for surprises. I once ran through the Program process blade with a team, only to have them come to the realization that … they weren’t really a Program! Which was actually a good thing as it helped avoid introducing a considerable amount of overhead, particularly in the area of program-level KPIs.

On Monday I attended a talk by Mike Rosen about successful Digital Transformation.  As always, Mike’s presentation was very insightful.  One of many points that he made is that most organizations are struggling with the current talent shortage, something that in my experience has been a problem worldwide for several years and will continue to be so.

There are many reasons for why there is a talent shortage:

1. It takes years to master your craft.  In the distant past, the 1980s, I had to go to university for four years to become a junior programmer.  Then I had to work for years to gain the experience and skills to become a developer. And things are certainly more complex now than it was back then. My point is that it takes years, not days or weeks, to gain the skills and experience that organizations are looking for.
2. Organization’s planning horizons are too short. Demand has outstripped supply for many years within IT, and we’re clearly seeing this within the agile space. Having said that, the growth of IT has followed a steady and predictable curve for years that organizations could have planned for, and more importantly executed on, had they chosen to. But, the multi-year time frame to grow talent doesn’t mesh well with the quarterly-results focus of most organizations.
3. Organizations have cut back on training. In the 1980s and most of the 1990s it was common for IT people to be given two or more weeks of training every year.  This was slowly whittled away down now many people are lucky to be given the funding to attend a single workshop or one-day conference each year.  Yes, there are some great resources online that can provide a substitute for training, so that’s bit of help. Unfortunately most organizations expect people to learn on their own time.  Some people do in fact do that, but many do not. I regularly meet people at conferences who have taken a vacation day and spent their own money to attend the event – it’s laudable that they’re doing this, but incredibly frustrating that they’ve had to resort to doing so. My point is that if your organization has not been investing in its people it seems disingenuous for it to complain about the lack of skilled people available to it.
4. Apprenticing is virtually non-existent in IT. Many other professions have a culture of apprenticing to help bring new people in and help them to gain the skills and experience required. Although I have seen several organizations attempt to institute a mentorship program, and have seen limited successes at doing so, I have never run into an organization with active apprenticing. Non-solo work practices such as pair programming, mob programming, and modeling with others are arguably the start at apprenticing practices within the agile space so perhaps this is how it begins in IT.  Time will tell.

Here are a few things that your organization can do to start addressing its talent shortage:

1. Become an attractive place to work. It’s important to note that attractive organizations, the ones that are known for doing interesting work and for providing their people with autonomy to learn and to choose their way of working (WoW), don’t seem to have problems attracting and retaining good people.  Granted, this doesn’t grow the pool of talented people but it does help to address your challenge.
2. Start investing in your people long term. If you want talented people in your organization you need to grow and nurture them yourself.  You can do so by providing coaching, mentoring, training, and education opportunities.  You can support the creation of communities of practice (CoPs), also called guilds, and create centres of excellence (CoEs).  Motivating teams to experiment with non-solo work practices such as pairing and mobbing will enable them to learn from each other.  And of course all of these strategies require investment of time and money.
3. Become a learning organization. If you want people to grow their skills you have to allow them to do so.  Allow them to experiment with new ways of working (WoW), recognizing that some of those experiments will “fail” and you’ll discover what doesn’t work for you in your context. Also recognize that individuals and teams have to be allowed to choose their WoW, and to evolve it as they learn, so that they can improve.

The primary reason why there is a shortage of talented people is because organizations are underinvesting in the learning paths of their staff.  If you want talented people you’re going to have to help create them.

Related reading:

• The People Management process blade
• The Grow Team Members process goal