At the Agile 2019 conference in DC I facilitated a workshop with about 70 people where we explored the topic of how do you coach an agile data warehousing (DW)/business intelligence (BI) team. To do this we worked through four issues:
The basic strategy was to introduce the issues to the class one at a time, then at their tables they would discuss the issue and write up to five ideas on sticky notes, then we’d share the ideas. Pictures of the flipcharts for each issue follow below. After the groups shared their ideas I then shared my thoughts with the class.
Issue #1: What Challenges Do You Face Coaching DW/BI Teams?
As you can see the class identified a lot of the classic issues that agile coaches face in general, such as trust issues, the teams being management-driven instead of self organizing, lack of agile skills within the team, cross-team dependencies, and being overwhelmed with work. Certainly there were DW/BI flavours of common problems, such as how to do vertical slices of DW/BI functionality and which lifecycle (agile, lean, CD, …) to follow. But there were also DW/BI specific issues, such as lack of access to data sources, knowing the actual data, and DW/BI architecture and design strategies. These DW/BI specific issues are where agile coaches tend to get hung up.
In my discussion of the challenges that we face when coaching agile DW/BI teams I shared my thoughts on the cultural impedance mismatch that exists between the agile and data communities. This mismatch makes it a bit more difficult to engage with data teams as opposed to application development teams. I also shared results of studies (2009, 2013,2016, 2018) around data quality challenges and practices – it is certainly common for teams to have to deal with technical debt, but data technical debt is both different in nature than code quality debt and the traditional data culture has led them down a very questionable (read dysfunctional) path regarding quality practices.
Issue #2: What Skills/Knowledge Does an Agile DW/BI Coach Require?
The second issue that we explored was what skills/knowledge does an agile DW/BI coach need. Once again the groups identified both classic agile coaching ideas as well as DW/BI specific ideas. Clearly you need coaching skills in order to coach a DW/BI team. But you also need to be knowledgeable about critical skills such as data modeling, data analysis, database testing, database refactoring, and others. You might not be an expert at these things, but you need to know of them and be able to guide the team in their adoption. You’ll also need to be able to speak intelligently about why some of the traditional strategies that they likely hold near and dear to their hearts (remember the cultural impedance mismatch) need to be abandoned for better, more effective strategies.
In my discussion I overviewed the “agile database techniques stack,” a collection of agile strategies and practices for database development. The stack is:
As you can see, this list of techniques is fairly common from an agile point of view, albeit the corresponding data(base) versions of those techniques. The point is that the techniques exist that enable data professionals to work in an agile, and far more effective, manner. As a coach you will need to be aware of these strategies and be able to help your DW/BI team adopt them. And of course there are agile data management strategies that you need to be aware of as well.
Issue #3: What Strategies Should You Use To Engage Successfully With An Agile DW/BI Team?
The groups identified a collection of great strategies for engaging with DW/BI teams. Once again there were a lot of standard coaching strategies, a DW/BI team is still a group of people after all, but there was also a focus on strategies to address the DW/BI challenges identified earlier.
The discussion that followed the sharing of the stickies a very interesting point was brought up. I had earlier stated that my experience with coaching DW/BI teams was that it was different than coaching other types of teams, mostly because of the cultural impedance mismatch. A handful of agile DW/BI coaches in the audience disagreed with that, pointing out that the critical issue was gaining the trust and respect of the team at the start. This is true of any team, and certainly true of DW/BI teams. To do this you need to understand and appreciate the issues that they deal with and be able to show that you know how to guide them through addressing their issues. You might not be an expert in the techniques of the agile database technique stack, or other important agile data techniques, but you do know of them and can help the team learn them. So yes, engaging with an agile DW/BI team is no different on the surface, but it does require the coach to have different skills and knowledge than what your typical agile coach has.
Issues #4: What Are The Qualities You Should Look For In An Agile DW/BI Coach?
For this exercise I pretty much asked the groups to put together a list of qualities for a job ad for an Agile DW/BI coach. This is what they came up with.
Here are our main learnings regarding coaching an agile DW/BI team:
I recently ran into The Oath for an Agile Coach. There are clearly some great ideas in the oath and it would be hard to argue that you wouldn’t want to adopt the advice contained within it. So I won’t do that. However, I do feel that there are some serious challenges surrounding the oath but that with a bit of hard work we could do better.
Some Great Ideas Here
Frankly, what’s not to like? The oath promotes the idea that coaches should do no harm, that they’re guests, that they should respect learnings, that they value discretion, and many other wonderful philosophies. Several of them are arguably a bit naive, for example:
It’s clear to me that a lot of smart people have put a lot of effort into the oath, that they’ve thought it through, and are honestly trying to make things better. I also believe that this is a step in the right direction, although at the time of this writing there are some serious challenges surrounding it that can and should be addressed.
A Few Serious Challenges
First and foremost, we should give the authors of the oath the benefit of the doubt and assume that they aren’t doing the things I’m about to describe on purpose. Although what I have to say is harsh, I honestly believe that the authors have their hearts in the right place but have not thought through the implications of what they’ve started. So here goes.
The oath is deceptive and as a result possibly unethical. The reason why I say this is that they claim to have based the coach’s oath on the The Hippocratic Oath (which I’m sure they’ve actually done). The problem is that they’ve merely skimmed the surface of the Hippocratic Oath, lifting ideas such as “First, do no harm” (which the oath doesn’t actually say, that’s the Hollywood interpretation of it) without also adopting the context in which the Hippocratic Oath is taken. This is important. New medical practitioners, after years of training, are asked to take the oath, or something similar, by medical schools. These schools are governed and the medical professionals themselves are governed. Control mechanisms are in place to ensure that the people who take the oath know what they’re doing and work in a trustworthy manner. Therein lies the rub – no such governance exists for agile coaching and I suspect the vast majority of agile coaches would chaff at the suggestion.
To see why this is an issue consider the following example. I have no medical training or background, with the exception of taking a few first aid courses over the years. Come to think of it, by agile standards I have more than enough medical training to be considered a Certified Surgery Master (CSM), so it’s all good. I have just now recited the Hippocratic Oath and have pledged to abide by it. As a result I now feel that I am qualified to offer plastic surgery procedures as I’ve heard that this is a lucrative business to be in. If you would like a face lift, liposuction, or augmentation of a body part please contact me to arrange a procedure. You can trust me because I’ve recited the Hippocratic Oath and I’m a CSM. What? You’re not interested? I’ve pledged to do no harm, so you can trust me.
I think that you inherently know it would be a bad thing for me to perform plastic surgery on you. I’m obviously not qualified. Therein lies the rub. I could easily advertise that I’ve pledged the oath, tell people about my CSM credentials, and make it sound like I’m qualified, particularly to people who don’t have much of a background in agile. In fact, recently in Toronto, a 19-year old woman did something very similar to this and as you’d expect it didn’t work out well for the recipients of her surgery endeavours.
By claiming that the agile coach’s oath is based on the Hippocratic Oath the authors are taking advantage of something called “prestige association.” The Hippocratic Oath is prestigious – the people who pledge it have to work very hard to be asked to pledge it and are subsequently held to its high standards throughout their careers. By explicitly associating the agile coaching oath with the Hippocratic Oath the prestige of the latter is conferred to the former. This is deceptive at best and unethical at worst. I believe we can be better than this.
How We Can Do Better
It isn’t appropriate to complain about the Agile Coach’s Oath without also providing some possible ways to fix it. Here are my initial thoughts:
I believe the people who developed The Oath for an Agile Coach have good intentions. They’ve gotten a great start on an interesting and potentially valuable idea. But, they need to follow through and make it something real if they really want this oath to be meaningful. I hope they choose to build a vibrant community that does exactly that. Time will tell.
When your organization chooses to transition to more agile and lean ways of working you quickly discover that this effort needs to address all aspects of your organization, not just your solution delivery teams. Many transformation efforts invest in agile team coaches, which is a very good thing to do, but will often shortchange other areas of coaching in the belief that they’ll figure it out on their own. It may work out that way, but even when it does this is an expensive, slow, and error-prone approach. In our experience it’s far better to get help from an experienced Enterprise Coach.
An Enterprise Coach coaches “beyond the team” to help senior managers and leaders to understand and adopt an agile and lean mindset. As you will soon see, this requires a similar yet different skill set than what is required for team coaching. In this blog we work through three key concepts:
Types of Agile Coaches
As you see in the following diagram we like to distinguish between several types of coaches:
Skills of An Enterprise Coach
The skills of an Enterprise Coach include:
Supporting Other Coaches
Enterprise Coaches support other coaches in several important ways:
There is of course a lot more to agile coaching that what is covered in this short blog. Our goal with this writing was to overview the role of Enterprise Coach and show how it fits into the overall scheme of things.
A common question that people ask is how does the adoption of agile within a team affect its productivity? The answer to this question will vary by team, but there are several common patterns that we’ve seen over time. In this blog we explore:
What Does Increased Productivity Mean to You?
Productivity is defines various measures of the efficiency of production, and is calculated as
Value of Output divided by Cost of Input
The implication of this calculation is that there is flexibility in the way that we can increase the productivity of a team:
Remember that context counts – each team will choose the most appropriate way for them to increase their productivity. Having said that, a common result of a team adopting agile is to incrementally deliver value more often.
Agile Adoption Patterns
The following diagram overviews three common patterns when it comes to productivity change when teams adopt agile. You’ve likely seen simpler versions of this diagram elsewhere that only show the dark green line, but our experience is that’s only part of the story. You can see in all three cases that the adoption of agile results in an initial productivity loss on a team – this reflects the reality that with any type of change it will take time for a team to learn the new strategy, to identify how it fits into their environment, and to learn the new requisite skills and behaviours.
The three patterns, from least desirable to most desirable, are:
What Are the Key Milestones to Look For?
There are three key milestone points on the successful paths that you should watch out for:
How Can You Improve Your Chance of Success?
There are several strategies that you can employ to increase your chance of successfully adopting agile and shifting to a continuous improvement culture within your team:
How Do You Know That Your Productivity Actually Improved?
Although the chart above intuitively makes sense, how do you know that your productivity has actually increased? To definitively answer this question you need to determine what productivity means to you, what the productivity level of the team currently is, and then continue to measure the level of productivity over time. This strategy tends to fall apart because few organizations know how they want to measure team productivity and fewer yet have actual measures in place. This of course is particularly vexing when senior management still requires you to prove that your productivity has increased as the result of your agile adoption efforts. Luckily there are ways to measure the change in productivity even when you don’t know what the baseline productivity level currently is. We’ll address this topic in a future blog.
Agile is About More Than Productivity Improvement
There are many benefits to agile, improving team productivity being just one of them. Potential benefits, some of which lead to greater productivity, include:
I was hoping to come up with a pithy, short answer to this question but the only thing that I can come up with is “people.” The not-so-pithy answer is that there is no sort of agreement around what it means to be a “qualified agile coach”, the people hiring coaches aren’t thinking things through in many cases, and the agile community suffers from a myriad of integrity challenges when it comes to professionalism. In this blog I work through the following ideas:
Why is There A Dearth of Qualified Agile Coaches?
Let’s answer this in two parts: Why is there a dearth of agile coaches and why are there so many unqualified coaches available? The first question is very easy to answer. The demand for agile coaches far outstrips the supply. The adoption rate for agile has been growing steadily since 2001, hence the growing demand. As you’ll see later in this blog, it takes years to grow good coaches. As a result there is little hope for the supply to catch up with demand any time soon.
The second question, why are there so many unqualified coaches available, is easy but uncomfortable to answer. In general we have systemic challenges in the IT industry and in many ways we’ve managed to exacerbate these problems within the agile community. Some of the challenges within the IT community include:
Then we have the agile community, with its various certification training scams. You can become a certified master after staying awake during a two-day workshop and passing an online test that almost nobody fails. To put this into context, a Starbucks barista, the kid who pours your morning coffee, get’s three days of training before being let loose on customers. Yet it somehow makes sense that someone with 50% less training becomes the lead of a software development team? Really? Another example: Someone can become a scaling program consultant after attending a four-day workshop, and worse yet are now “qualified” to teach a two-day workshop to others so as to impart their vast agile scaling knowledge upon them. Amazingly, because of the demand by companies desperate to hire agile-skilled people, the demand for these “designations” is incredible (shameful would be a more appropriate word).
In practice many agile designations are little more than “participation ribbons”, yet most organizations take them seriously often either because they don’t realize how trivial they are to earn or because they’ve given up expecting any better from agilists. Is it any surprise that it’s hard to find qualified coaches when we’ve watered things down so much?
Sports Coaches as an Example
Coaching is very common in sports and with the exception of “pick up” games few sports teams are without a coach. In fact, serious sports teams tend to have several coaches, typically lead by a head coach. In professional sports coaches are paid significant salaries, sometimes millions of dollars a year, as coaching is perceived to be a critical success factor. It makes sense to look at sports coaching works to see how agile coaching might work.
Most sports coaches are former players. They’ve typically played for years, and sometimes decades, having been coached themselves all along the way. They’ll often start off as children, in Canada it’s common for kids to start learning to skate and play hockey at the age of two, being coached and drilled in basic skills and knowledge for years. They also gain practical experience playing games. Most kids drop out eventually, although many still play their sport (be it hockey, football, cricket, baseball, …) well into middle age. And some decide to stay in the sport, but make the shift from being a player into being a coach.
The transition to becoming a sports coach generally isn’t easy. There are three common strategies for this:
So, what are some important observations we can make out of all of this? First, sports coaches have deep skills and experience at the sports that they are coaching. Second, we expect this of them. Would you pay to have your child to be given skating lessons by a “Certified Skating Master” who had two days of training in the “skating mindset” and how to facilitate a handful of skating meetings? Of course you wouldn’t. Instead you’d want someone who had been skating for years, and better yet may have even been a competitor at some point in the past. Third, it takes years of apprenticing or training to become a good sports coach, not just several days in a certification workshop.
What Should We Expect From Agile Coaches?
Here is what we’ve found to be the critical success factors for agile coaches:
Our Solution: Certified Disciplined Agile Coaches (CDACs)
A fair question to ask is how do we deal with this in the Disciplined Agile (DA) space. We believe that it’s critical to your success to have qualified coaches so we’ve built a principled certification program based on the martial arts philosophy of Shu-Ha-Ri. Certifications must be earned and that takes time. The following diagram summarizes our strategy for how practitioners must earn DA certifications.
To become a Certified Disciplined Agile Coach (CDAC) you need to have at least five years of experience in agile (which is verified by the Disciplined Agile Consortium (DAC)), plus evidence that you’ve already been sharing your skills and knowledge (we call this give back), plus you must be a Certified Disciplined Agile Practitioner (CDAP). To become a CDAP you must have at least two years of proven agile experience (validated by DAC), have passed a comprehensive test of your agile knowledge, and already be a Certified Disciplined Agile Practitioner (CDA). To be a CDA you must have passed a comprehensive test of your knowledge and skills. So, this process of certification ensures that CDACs have comprehensive skills and knowledge in agile techniques, at least five years experiences in agile, and at least initial experience in coaching/teaching (the giveback component). Note: Not shown in the diagram above is Certified Disciplined Agile Instructor (CDAI), which you must be at least a CDAP and have proven ability to teach DA.
It isn’t easy to find qualified agile coaches, but then again it isn’t impossible either. Our hope is that this blog has provided you with some insight into what you should be looking for in a good agile coach. Anyone can put a shingle up and say that they’re an “agile coach”, but anyone who wants to say that they are a Certified Disciplined Agile Coach (CDAC) needs to have worked through a rigorous process to earn that qualification. CDACs have proven knowledge, experience, and give back. Why settle for less?