Since Disciplined Agile (DA) joined the PMI family in August 2019 we've gotten a collection of questions from people along the lines of "Why is there a difference between the advice in DA and PMI's advice?" So I thought I would write a few blogs examining why that is. This is the first.
There are several reasons why there are differences between existing DA and existing (non-DA) PMI materials:
They were created by different groups of people. It's natural to get different takes on a topic from different groups of people.
DA took on a broader scope than PMI traditionally has (until now). PMI has focused on project management and critical topics surrounding it such as program management, portfolio management, and governance (amongst others). That is the scope that PMI chose to focus on and has frankly done a very good job at doing so. The scope of DA, on the other hand, has been to address how to take an agile/lean approach to all aspects of an organization, including but not limited to management. This is a much broader scope than what PMI has taken on, until now. As a result DA addresses marketing, finance, enterprise architecture, operations, governance, software development, and many other process areas that are important to modern organizations. Why is this broader scope important to PMI? Because all of these areas need to be managed/led and governed. I believe there's an interesting implication there. ;-)
PMI has traditionally gone very deep into management and the governance of management activities. I'll let the great material in our standards and practice guides speak for itself. As Stan Lee was prone to say, 'Nuff said.
DA has traditionally taken a more holistic view. DA includes both what is being managed as well as the management/leadership of it. For example, consider The Standard for Program Management Fourth Edition. Where the existing PMI standard does a fantastic job of addressing the management aspects of a traditional program it doesn't go into critical "doing aspects" of programs such as how to address architecture, requirements, and quality activities (it does address planning and management though) for example. This isn't meant to be a criticism of the standard but merely an observation - When we (PMI) developed the standard our focus, and once again rightfully so, was on management and governance. It was not on the overall, holistic view of what occurs with a program. With DA we choose to take a more holistic view, as do agile frameworks such as SAFeR and LeSS amongst others, and go beyond management and governance.
My point is that there are very good reasons for the differences between what is in DA and what PMI has traditionally focused on. These differences are an important aspect of the value proposition of DA for PMI, and more importantly for our membership, because we can learn from these differences and then improve and grow based on those learnings. We're currently evolving DA based on the great material encompassed by the existing PMI standards and practice guides and our hope is that the existing PMI offerings will evolve to reflect Disciplined Agile ways of working (WoW) too.
In the next blog in this series I will do a deep dive into the differences between DA's take on Program Management and the PMI Program Management Standard. I suspect this will help to make some of the ideas in this blog more concrete and it will certainly make the opportunity before us a bit more explicit.
Kurt Cagle recently wrote an article for Forbes, entitled The End of Agile. Although Forbes’ regular Steve Denning responded effectively a few days later with Why Agile’s Future is Bright, I’d like to chime in with several thoughts beyond what Steve has offered. Here they are:
Look beyond Scrum. Kurt started out with a tongue-in-cheek description of an agile team, but it was really a description of a small team at the very beginnings of learning Scrum. This is the classic “Scrum = Agile” mistake that many people make, and it plays well to people who only have a passing understanding of how agile works in practice. We recommend that you look at teams that are succeeding with agile, rather than teams that are clearly struggling. Better yet, let’s help the teams who are struggling — rather than making fun of them.
Beware of agile purists. Kurt’s observation that agile has become a religion wasn’t far off. It might be more accurate to say that there are Agile purists out there, people who often prove to only have experience with applying agile in straightforward situations — rather than in the enterprise situations that Kurt describes. But there are also pragmatists out there in the Agile space (pragmatism is one of the seven Disciplined Agile principles) who are actively addressing how to apply agile and lean strategies in less-than-ideal situations. We have always recommended that you be open-minded and pragmatic, to do the best you can in the situation that you face and always strive to learn and improve.
Agile works in a very wide range of situations.There is ample research data and case studies showing that agile works in a wide range of situations. For example, in our 2016 Agility at Scale survey we found that organizations were in fact applying agile on large teams, in complex domains, using legacy technologies (including legacy data sources), in regulatory environments, and multi-organization situations including outsourcing. And we have data from other studies going back over a decade that show that Agile strategies have been applied beyond the “small teams with straightforward problems” scenario for quite a while. We recommend that you look around and see how agile is being applied in practice.
Scaling agile requires discipline. The article also states that agile doesn’t scale well to address big problems, yet many organizations are doing just that. To be fair to Kurt, this is a common misunderstanding that is a result of the Scrum = Agile mindset because Scrum purposefully doesn’t address architecture (amongst many topics). But other agile methods do — in particular Agile Modeling and Jim Coplien’s excellent work around Lean Architecture — strategies which are explicitly part of Disciplined Agile. In Disciplined Agile Delivery (DAD), we explicitly include initial architecture modeling early in a project via the Identify Architecture Strategy process goal, we show how to reduce architectural risk via the Prove Architecture Early process goal, and how to evolve architecture throughout Construction via the Produce a Potentially Consumable Solution process goal. DAD teams also have someone in an explicit Architecture Owner role who is responsible for guiding the team through architecture decisions. Furthermore, there is an explicit Enterprise Architecture process blade to support organization-level architecture issues. Finally, DA the idea that Architecture Owners and Enterprise Architects should work closely together. Our recommendation is to explicitly address architecture in your way of working, and this is something that Disciplined Agile provides clear, proven options for.
Data projects can be difficult. The same can be said of non-data projects too. When either domain complexity or technical complexity rise, or both, you need to become more disciplined in your approach to requirements and architecture modeling. As Cagle implies, data projects often suffer from domain complexity, this is particularly true for the enterprise data systems he mentions. Similarly data projects tend to suffer from technical complexity in that they need to integrate data from many disparate sources that often use different technologies, apply different semantics, and are accessed via different strategies. Once again, the context-sensitive choice-driven strategy promoted by Disciplined Agile wins the day over the prescriptive methods and frameworks Cagle appears to be familiar with. To confuse matters further, prescriptive frameworks such as Scrum and SAFe have virtually nothing to say about addressing data issues. Disciplined Agile, on the other hand, encompasses proven strategies from the Agile Data method which are critical to success on data projects. Our recommendation is to embrace the complexity that you face and to recognize that you will need to explicitly tailor your way of working (WoW) to address that complexity.
Agile is applicable to data projects…and may be the superior approach. I’m going to be blunt here – Kurt Cagle was completely and utterly wrong concerning the application of agile strategies to data projects. While traditional data professionals often prefer to do extensive data modeling up front, this strategy has been shown to be ineffective in practice. What does work well is an agile data modeling strategy where high-level modeling is performed early in a project and detailed data modeling performed as needed during construction. You can be very agile when supported by agile database practices such as database refactoring, automated database regression testing, and continuous database integration. And his claim that the focus on enterprise data is relatively new clearly doesn’t reflect the wealth of techniques around enterprise data modeling, master data management, and meta data management that the data community has been following since at least the mid-1990s. And yes, there are agile approaches to all of those practices. As an aside, I started writing this blog in the evenings while attending the World Wide Data Vault Conference in Hannover. Most of the presenters have discussed how they’re taking, or at least supporting agile — and often a Disciplined Agile — approaches on their data projects.
Many other claims the article makes are observably false. Contrary to what Kurt claims, there are many large and complex open source projects, and they clearly benefit from agile and lean strategies. Examples of such projects include Linux (operating system), Hadoop (big data processing), MongoDB (database), Numenta (machine learning), Angular (web application framework), and Docker (software infrastructure) to name but a few. But hey, it’s not as if any of those technologies have become mission critical for your organization. And his discussion of games development? Having actually consulted to several commercial game companies, I can safely say that they were all working in a very agile manner.
You must adapt your approach to address the context that you face. One process size does not fit all. Two of DA’s principles are Context Counts and Choice is Good – different teams in different situations will choose to work in different manners. If you want your teams to be effective then you have to allow them — and better yet, enablethem — to choose their own way of working (WoW).
Adapting your approach requires skill and experience. In some ways, Kurt is right – enterprise-class problems require an enterprise-class agile approach. But he was wrong in assuming that because people were struggling to apply undisciplined agile strategies to these situations that agile didn’t work; the real problem was that they didn’t know how to make it work. The fact is that others have figured these things out, and we’ve captured a lot of these strategies in PMI’s Disciplined Agile (DA) toolkit.
Let’s hope this is the end of undisciplined agile. But as Steve Denning points out, it certainly isn’t the end of agile. I suggest this could be an important beginning for Disciplined Agile approaches.
Although all of the presentations were great, I was particularly enthralled with Bill Inmon’s keynote on Thursday. As you may know Bill is the father of data warehousing and has written 59 (59!) books over his career, clearly putting me to shame. Bill shared some of his experiences in extracting information from text-based sources and described several stories doing so. One story focused on how his team had combined information culled from multiple text-based sources that together indicated that BP had a potential maintenance risk in their Gulf of Mexico operations. Sadly his warning was ignored and several months later BP had a catastrophic oil rig failure on Deepwater Horizon. Another story described how his team processed 5000 online postings from Nike customers and 5000 from Adidas customers. Their analysis indicated that while Adidas was a “normal company,” that on the other hand Nike had quality problems with their shoes. Although Bill contacted Nike to inform them, free of charge, of what he had discovered this advice was also ignored because Nike apparently already had consulting companies providing them with advice. A year later Nike suffered a $2 billion market capitalization loss when Zion Williamson’s sneaker exploded in a basketball game watched by over 100 million people. Another text analysis project led him to discover that airlines are consistently not well liked by their customers, revealing that Bill doesn’t always end up with earth-shattering revelations. Although the stories were interesting, Bill’s description of the techniques he was following and the challenges surrounding text-based data analytics were fascinating.
Data Vault 2 (DV2) is an extension to Inmon’s approach to data warehousing. Dan Lindstedt, the creator of DV2, worked for years with Bill. DV2 brings a lot of very practical strategies to data warehousing. Furthermore, a few years ago DV2 adopted Disciplined Agile Delivery (DAD) for its process which was one of the reasons why I suspect I was invited to speak at the conference.
Kudos to everyone who made the conference a success. I’m looking forward to next year.
We recently recorded a short discussion between Sunil Prashara, CEO of PMI; Dave Garrett, VP of Corporate Development and Innovation at PMI; and Mark Lines, co-creator of Disciplined Agile and now VP of Disciplined Agile at PMI. In the audio recording, which you can find at Quick Podcast on PMI’s Disciplined Agile Acquisition, they share their thoughts about four interesting questions:
How does Disciplined Agile go beyond agile software development?
At the Agile 2019 conference in DC I facilitated a workshop with about 70 people where we explored the topic of how do you coach an agile data warehousing (DW)/business intelligence (BI) team. To do this we worked through four issues:
What challenges do you face?
What skills/knowledge does an agile DW/BI coach require?
What strategies should you apply to engage with a DW/BI team?
What are the qualities you should look for in an agile DW/BI coach?
The basic strategy was to introduce the issues to the class one at a time, then at their tables they would discuss the issue and write up to five ideas on sticky notes, then we’d share the ideas. Pictures of the flipcharts for each issue follow below. After the groups shared their ideas I then shared my thoughts with the class.
Issue #1: What Challenges Do You Face Coaching DW/BI Teams?
As you can see the class identified a lot of the classic issues that agile coaches face in general, such as trust issues, the teams being management-driven instead of self organizing, lack of agile skills within the team, cross-team dependencies, and being overwhelmed with work. Certainly there were DW/BI flavours of common problems, such as how to do vertical slices of DW/BI functionality and which lifecycle (agile, lean, CD, …) to follow. But there were also DW/BI specific issues, such as lack of access to data sources, knowing the actual data, and DW/BI architecture and design strategies. These DW/BI specific issues are where agile coaches tend to get hung up.
Challenges commonly faced when coaching agile DW/BI teams (click to enlarge).
In my discussion of the challenges that we face when coaching agile DW/BI teams I shared my thoughts on the cultural impedance mismatch that exists between the agile and data communities. This mismatch makes it a bit more difficult to engage with data teams as opposed to application development teams. I also shared results of studies (2009, 2013,2016, 2018) around data quality challenges and practices – it is certainly common for teams to have to deal with technical debt, but data technical debt is both different in nature than code quality debt and the traditional data culture has led them down a very questionable (read dysfunctional) path regarding quality practices.
Issue #2: What Skills/Knowledge Does an Agile DW/BI Coach Require?
The second issue that we explored was what skills/knowledge does an agile DW/BI coach need. Once again the groups identified both classic agile coaching ideas as well as DW/BI specific ideas. Clearly you need coaching skills in order to coach a DW/BI team. But you also need to be knowledgeable about critical skills such as data modeling, data analysis, database testing, database refactoring, and others. You might not be an expert at these things, but you need to know of them and be able to guide the team in their adoption. You’ll also need to be able to speak intelligently about why some of the traditional strategies that they likely hold near and dear to their hearts (remember the cultural impedance mismatch) need to be abandoned for better, more effective strategies.
Skills/knowledge required of an agile DW/BI coach (click to enlarge).
In my discussion I overviewed the “agile database techniques stack,” a collection of agile strategies and practices for database development. The stack is:
As you can see, this list of techniques is fairly common from an agile point of view, albeit the corresponding data(base) versions of those techniques. The point is that the techniques exist that enable data professionals to work in an agile, and far more effective, manner. As a coach you will need to be aware of these strategies and be able to help your DW/BI team adopt them. And of course there are agile data management strategies that you need to be aware of as well.
Issue #3: What Strategies Should You Use To Engage Successfully With An Agile DW/BI Team?
The groups identified a collection of great strategies for engaging with DW/BI teams. Once again there were a lot of standard coaching strategies, a DW/BI team is still a group of people after all, but there was also a focus on strategies to address the DW/BI challenges identified earlier.
Strategies to engage successfully with a DW/BI team (click to enlarge).
The discussion that followed the sharing of the stickies a very interesting point was brought up. I had earlier stated that my experience with coaching DW/BI teams was that it was different than coaching other types of teams, mostly because of the cultural impedance mismatch. A handful of agile DW/BI coaches in the audience disagreed with that, pointing out that the critical issue was gaining the trust and respect of the team at the start. This is true of any team, and certainly true of DW/BI teams. To do this you need to understand and appreciate the issues that they deal with and be able to show that you know how to guide them through addressing their issues. You might not be an expert in the techniques of the agile database technique stack, or other important agile data techniques, but you do know of them and can help the team learn them. So yes, engaging with an agile DW/BI team is no different on the surface, but it does require the coach to have different skills and knowledge than what your typical agile coach has.
Issues #4: What Are The Qualities You Should Look For In An Agile DW/BI Coach?
For this exercise I pretty much asked the groups to put together a list of qualities for a job ad for an Agile DW/BI coach. This is what they came up with.
Potential qualities of an agile DW/BI coach (click to enlarge).
Our Learnings
Here are our main learnings regarding coaching an agile DW/BI team:
To effectively engage with a DW/BI team, you need to gain their trust and respect – to do that you need to show that you understand the challenges that they face and can help them to address them
To do so can be more challenging than with an application team due to the cultural impedance mismatch, the nature of data-oriented technical debt, and the lack of modern development skills within the data community – all of these challenges point to a greater than normal need for coaching on such teams
