Kurt Cagle recently wrote an article for Forbes, entitled The End of Agile. Although Forbes’ regular Steve Denning responded effectively a few days later with Why Agile’s Future is Bright, I’d like to chime in with several thoughts beyond what Steve has offered. Here they are:
Let’s hope this is the end of undisciplined agile. But as Steve Denning points out, it certainly isn’t the end of agile. I suggest this could be an important beginning for Disciplined Agile approaches.
Earlier today I keynoted at the World Wide Data Vault Consortium European conference in Hannover Germany. I presented an overview of Disciplined Agile, some of the challenges that organizations are experiencing in their agile transformations, and how their teams can improve their way of working (WoW) via Guided Continuous Improvement (GCI).
Although all of the presentations were great, I was particularly enthralled with Bill Inmon’s keynote on Thursday. As you may know Bill is the father of data warehousing and has written 59 (59!) books over his career, clearly putting me to shame. Bill shared some of his experiences in extracting information from text-based sources and described several stories doing so. One story focused on how his team had combined information culled from multiple text-based sources that together indicated that BP had a potential maintenance risk in their Gulf of Mexico operations. Sadly his warning was ignored and several months later BP had a catastrophic oil rig failure on Deepwater Horizon. Another story described how his team processed 5000 online postings from Nike customers and 5000 from Adidas customers. Their analysis indicated that while Adidas was a “normal company,” that on the other hand Nike had quality problems with their shoes. Although Bill contacted Nike to inform them, free of charge, of what he had discovered this advice was also ignored because Nike apparently already had consulting companies providing them with advice. A year later Nike suffered a $2 billion market capitalization loss when Zion Williamson’s sneaker exploded in a basketball game watched by over 100 million people. Another text analysis project led him to discover that airlines are consistently not well liked by their customers, revealing that Bill doesn’t always end up with earth-shattering revelations. Although the stories were interesting, Bill’s description of the techniques he was following and the challenges surrounding text-based data analytics were fascinating.
Data Vault 2 (DV2) is an extension to Inmon’s approach to data warehousing. Dan Lindstedt, the creator of DV2, worked for years with Bill. DV2 brings a lot of very practical strategies to data warehousing. Furthermore, a few years ago DV2 adopted Disciplined Agile Delivery (DAD) for its process which was one of the reasons why I suspect I was invited to speak at the conference.
Kudos to everyone who made the conference a success. I’m looking forward to next year.
At the Agile 2019 conference in DC I facilitated a workshop with about 70 people where we explored the topic of how do you coach an agile data warehousing (DW)/business intelligence (BI) team. To do this we worked through four issues:
The basic strategy was to introduce the issues to the class one at a time, then at their tables they would discuss the issue and write up to five ideas on sticky notes, then we’d share the ideas. Pictures of the flipcharts for each issue follow below. After the groups shared their ideas I then shared my thoughts with the class.
Issue #1: What Challenges Do You Face Coaching DW/BI Teams?
As you can see the class identified a lot of the classic issues that agile coaches face in general, such as trust issues, the teams being management-driven instead of self organizing, lack of agile skills within the team, cross-team dependencies, and being overwhelmed with work. Certainly there were DW/BI flavours of common problems, such as how to do vertical slices of DW/BI functionality and which lifecycle (agile, lean, CD, …) to follow. But there were also DW/BI specific issues, such as lack of access to data sources, knowing the actual data, and DW/BI architecture and design strategies. These DW/BI specific issues are where agile coaches tend to get hung up.
In my discussion of the challenges that we face when coaching agile DW/BI teams I shared my thoughts on the cultural impedance mismatch that exists between the agile and data communities. This mismatch makes it a bit more difficult to engage with data teams as opposed to application development teams. I also shared results of studies (2009, 2013,2016, 2018) around data quality challenges and practices – it is certainly common for teams to have to deal with technical debt, but data technical debt is both different in nature than code quality debt and the traditional data culture has led them down a very questionable (read dysfunctional) path regarding quality practices.
Issue #2: What Skills/Knowledge Does an Agile DW/BI Coach Require?
The second issue that we explored was what skills/knowledge does an agile DW/BI coach need. Once again the groups identified both classic agile coaching ideas as well as DW/BI specific ideas. Clearly you need coaching skills in order to coach a DW/BI team. But you also need to be knowledgeable about critical skills such as data modeling, data analysis, database testing, database refactoring, and others. You might not be an expert at these things, but you need to know of them and be able to guide the team in their adoption. You’ll also need to be able to speak intelligently about why some of the traditional strategies that they likely hold near and dear to their hearts (remember the cultural impedance mismatch) need to be abandoned for better, more effective strategies.
In my discussion I overviewed the “agile database techniques stack,” a collection of agile strategies and practices for database development. The stack is:
As you can see, this list of techniques is fairly common from an agile point of view, albeit the corresponding data(base) versions of those techniques. The point is that the techniques exist that enable data professionals to work in an agile, and far more effective, manner. As a coach you will need to be aware of these strategies and be able to help your DW/BI team adopt them. And of course there are agile data management strategies that you need to be aware of as well.
Issue #3: What Strategies Should You Use To Engage Successfully With An Agile DW/BI Team?
The groups identified a collection of great strategies for engaging with DW/BI teams. Once again there were a lot of standard coaching strategies, a DW/BI team is still a group of people after all, but there was also a focus on strategies to address the DW/BI challenges identified earlier.
The discussion that followed the sharing of the stickies a very interesting point was brought up. I had earlier stated that my experience with coaching DW/BI teams was that it was different than coaching other types of teams, mostly because of the cultural impedance mismatch. A handful of agile DW/BI coaches in the audience disagreed with that, pointing out that the critical issue was gaining the trust and respect of the team at the start. This is true of any team, and certainly true of DW/BI teams. To do this you need to understand and appreciate the issues that they deal with and be able to show that you know how to guide them through addressing their issues. You might not be an expert in the techniques of the agile database technique stack, or other important agile data techniques, but you do know of them and can help the team learn them. So yes, engaging with an agile DW/BI team is no different on the surface, but it does require the coach to have different skills and knowledge than what your typical agile coach has.
Issues #4: What Are The Qualities You Should Look For In An Agile DW/BI Coach?
For this exercise I pretty much asked the groups to put together a list of qualities for a job ad for an Agile DW/BI coach. This is what they came up with.
Here are our main learnings regarding coaching an agile DW/BI team:
Lynn will soon be blogging about the results so I’m not going to dive into that here. I suspect that her blog post will be very interesting.
What I’d like to do here is share a few thoughts about what I observed:
I’m very happy to see that Lynn is actively trying to bridge the agile and data communities, helping us to learn from each other. This is something she’s been doing for years and doing it quite well. My experience is that both communities would benefit greatly from more collaboration with each other.
There are several strategies that you can choose to employ with vertically slicing the requirements for a DW/BI solution. These strategies are described in the following table. There are example stories for each strategy as well as some advice for when to apply each strategy.
Table 1. Vertical slicing strategies for a DW/BI solution.
There are several interesting things about the stories in the table:
I’ve written a detailed explanation of vertical slicing for a DW/BI solution, and of course there is a wealth of information about agile database techniques in general for those of you interested in greater detail. You may also find our one-day Disciplined Agile Data Warehousing/Business Intelligence Workshop to be a valuable learning experience too.