Disciplined Agile

To achieve organizational efficiency and success, consider improving the flow of value delivery in your organization. What are some considerations for your next leadership or team retrospective in order to optimize the flow of work?

In the context of this write-up, the flow of work being referring to is from Idea to Implementation of a consumable product increment. In order to visualize the flow, consider using process mapping or value stream mapping to identify all the steps from when an idea is brought up, through to approval of the idea. Steps to capture may include things like securing funding, identifying the team(s), prioritization, construction phase steps and releasing the increment of value.

Flow Considerations

Three areas to consider include Alignment, Dependencies and Prioritization. There are other considerations which impact the flow of work in your organization, but these are the three I often look at first. Here are some considerations/questions to ask your team and/or leaders when looking to improve flow.

Alignment

How might alignment help flow? Alignment is the secret sauce to achieving better business performance. Alignment will help increase the speed to decision making and decrease wasted resources waiting for decisions, re-hashing discussions, churn and effort working on lower value work. The Disciplined Agile principle of Enterprise Awareness helps leaders understand and work from the same playbook, the same strategy and desired outcomes for the organization. Alignment to overall objectives will help to drive the right values, practices and behaviors to achieve organizational success. Team alignment to the stakeholder vision and iteration goals will help the team remove work not related to the goal or lower value items not which does not help achieve the stakeholder vision.

Dependencies

There are several considerations for dependencies. Are there inter-team dependencies present in the delivery of a consumable increment? Where do items span multiple teams? Can these dependencies be measured in wait time or other impacts? How are the teams structured (business silos, technology focus vs cross-functional whole team focused on a product)? Do your team members have the necessary permissions in the systems in order to complete the construction and release of the product increment? Can team members just do the work without waiting for decisions or approvals?

Prioritization

How does priority impact flow to your team? What or who prioritizes the work? Is it a lengthy process with multiple people or a quick decision with an authorized person? How many steps are in the re-prioritization process? Is it nimble enough to quickly and efficiently respond to change? Are stakeholders going to the product owner or directly to team members? If they are going directly to team members, why? How are the team members responding? Are they honoring the team’s definition of ready before bringing work into an iteration? If you have a product owner, are they negotiating with the stakeholders to place their item in a future iteration, or is it being added to an in-flight iteration? Does your product owner have a consistent method of assessing value/risk? Is this a transparent process to the stakeholders? Do the stakeholders understand the stakeholder vision and why their item is higher or lower in priority at any given time?

Improving Flow

For more information on any of these three considerations, please refer to the Disciplined Agile toolkit for insights and choices to help guide an improvement to flow. As it relates specifically to prioritization, check out the Address Changing Stakeholder Needs process goal and consider what to prioritize, who will prioritize and how to prioritize. There may be many inputs from customer feedback, technical debt, stakeholder vision of the product etc. to consider. How we prioritize may first start with looking at the scope and intent of the present phase of delivery. Are you looking at mitigating risk? Building a strong yet flexible foundation? Delivering a high valuable consumable product to delight your customer? Understanding and aligning on this intent, perhaps using a stakeholder vision canvas can help to establish a prioritization process. Empower the product owner, architecture owner and team members to use the vision and process transparently in discussion with the stakeholders so they can provide input and negotiate how their item fits/or doesn’t with the current phase of delivery. Lastly, use backlog refinement sessions (also known as look-ahead modeling) as a conduit to get the stakeholders together in a room to discuss the next highest priority items.

Measuring Flow

What might your team use to measure flow?  A few measures to start with:

1. Decision turn-around time. How long is the team waiting for an approval? Can this be shortened or eliminated?
2. Technical debt. How do I raise the issue of technical debt to ensure it is prioritized alongside value? Can you measure the cost of support? How many workarounds exist in production? What are the costs of the workarounds to the business? Is the current technology easy to update and release for business innovation? What is the cost of not innovating or releasing something your competitor has?
3. Lead and cycle times.  Which processes or activities are heavier steps that are candidates for optimization?  What is the average time it takes to deliver a small sized user story from idea to implementation?
4. Process time. How long does the prioritization process itself take, before an item even gets to a delivery team? Starting with the longest steps, what can be done to shorten or eliminate these steps?

Hopefully, this summary has given you some things to consider in order to Optimize Flow. What are key areas you look at to optimize flow? What measures does your team use? Please add them in the comments section below.

Lisa Lueck, CDAC

Kurt Cagle recently wrote an article for Forbes, entitled The End of Agile. Although Forbes’ regular Steve Denning responded effectively a few days later with Why Agile’s Future is Bright, I’d like to chime in with several thoughts beyond what Steve has offered.  Here they are:

1. Look beyond Scrum. Kurt started out with a tongue-in-cheek description of an agile team, but it was really a description of a small team at the very beginnings of learning Scrum.  This is the classic “Scrum = Agile” mistake that many people make, and it plays well to people who only have a passing understanding of how agile works in practice. We recommend that you look at teams that are succeeding with agile, rather than teams that are clearly struggling.  Better yet, let’s help the teams who are struggling — rather than making fun of them.
2. Beware of agile purists. Kurt’s observation that agile has become a religion wasn’t far off.  It might be more accurate to say that there are Agile purists out there, people who often prove to only have experience with applying agile in straightforward situations — rather than in the enterprise situations that Kurt describes.  But there are also pragmatists out there in the Agile space (pragmatism is one of the seven Disciplined Agile principles) who are actively addressing how to apply agile and lean strategies in less-than-ideal situations.  We have always recommended that you be open-minded and pragmatic, to do the best you can in the situation that you face and always strive to learn and improve.
3. Agile works in a very wide range of situations.There is ample research data and case studies showing that agile works in a wide range of situations.  For example, in our 2016 Agility at Scale survey we found that organizations were in fact applying agile on large teams, in complex domains, using legacy technologies (including legacy data sources), in regulatory environments, and multi-organization situations including outsourcing.  And we have data from other studies going back over a decade that show that Agile strategies have been applied beyond the “small teams with straightforward problems” scenario for quite a while.  We recommend that you look around and see how agile is being applied in practice.
4. Scaling agile requires discipline. The article also states that agile doesn’t scale well to address big problems, yet many organizations are doing just that.  To be fair to Kurt, this is a common misunderstanding that is a result of the Scrum = Agile mindset because Scrum purposefully doesn’t address architecture (amongst many topics). But other agile methods do — in particular Agile Modeling and Jim Coplien’s excellent work around Lean Architecture — strategies which are explicitly part of Disciplined Agile.  In Disciplined Agile Delivery (DAD), we explicitly include initial architecture modeling early in a project via the Identify Architecture Strategy process goal, we show how to reduce architectural risk via the Prove Architecture Early process goal, and how to evolve architecture throughout Construction via the Produce a Potentially Consumable Solution process goal.  DAD teams also have someone in an explicit Architecture Owner role who is responsible for guiding the team through architecture decisions.  Furthermore, there is an explicit Enterprise Architecture process blade to support organization-level architecture issues.  Finally, DA the idea that Architecture Owners and Enterprise Architects should work closely together.  Our recommendation is to explicitly address architecture in your way of working, and this is something that Disciplined Agile provides clear, proven options for.
5. Data projects can be difficult. The same can be said of non-data projects too. When either domain complexity or technical complexity rise, or both, you need to become more disciplined in your approach to requirements and architecture modeling.  As Cagle implies, data projects often suffer from domain complexity, this is particularly true for the enterprise data systems he mentions. Similarly data projects tend to suffer from technical complexity in that they need to integrate data from many disparate sources that often use different technologies, apply different semantics, and are accessed via different strategies. Once again, the context-sensitive choice-driven strategy promoted by Disciplined Agile wins the day over the prescriptive methods and frameworks Cagle appears to be familiar with.  To confuse matters further, prescriptive frameworks such as Scrum and SAFe have virtually nothing to say about addressing data issues.  Disciplined Agile, on the other hand, encompasses proven strategies from the Agile Data method which are critical to success on data projects.  Our recommendation is to embrace the complexity that you face and to recognize that you will need to explicitly tailor your way of working (WoW) to address that complexity.
6. Agile is applicable to data projects…and may be the superior approach. I’m going to be blunt here – Kurt Cagle was completely and utterly wrong concerning the application of agile strategies to data projects.  While traditional data professionals often prefer to do extensive data modeling up front, this strategy has been shown to be ineffective in practice. What does work well is an agile data modeling strategy where high-level modeling is performed early in a project and detailed data modeling performed as needed during construction.  You can be very agile when supported by agile database practices such as database refactoring, automated database regression testing, and continuous database integration. And his claim that the focus on enterprise data is relatively new clearly doesn’t reflect the wealth of techniques around enterprise data modeling, master data management, and meta data management that the data community has been following since at least the mid-1990s. And yes, there are agile approaches to all of those practices.  As an aside, I started writing this blog in the evenings while attending the World Wide Data Vault Conference in Hannover.  Most of the presenters have discussed how they’re taking, or at least supporting agile — and often a Disciplined Agile — approaches on their data projects.
7. Many other claims the article makes are observably false. Contrary to what Kurt claims, there are many large and complex open source projects, and they clearly benefit from agile and lean strategies.  Examples of such projects include Linux (operating system), Hadoop (big data processing), MongoDB (database), Numenta (machine learning), Angular (web application framework), and Docker (software infrastructure) to name but a few.  But hey, it’s not as if any of those technologies have become mission critical for your organization.  And his discussion of games development?  Having actually consulted to several commercial game companies, I can safely say that they were all working in a very agile manner.
8. You must adapt your approach to address the context that you face. One process size does not fit all. Two of DA’s principles are Context Counts and Choice is Good – different teams in different situations will choose to work in different manners.  If you want your teams to be effective then you have to allow them — and better yet, enablethem — to choose their own way of working (WoW).
9. Adapting your approach requires skill and experience. In some ways, Kurt is right – enterprise-class problems require an enterprise-class agile approach. But he was wrong in assuming that because people were struggling to apply undisciplined agile strategies to these situations that agile didn’t work; the real problem was that they didn’t know how to make it work.  The fact is that others have figured these things out, and we’ve captured a lot of these strategies in PMI’s Disciplined Agile (DA) toolkit.

Let’s hope this is the end of undisciplined agile. But as Steve Denning points out, it certainly isn’t the end of agile.  I suggest this could be an important beginning for Disciplined Agile approaches.

Earlier today I keynoted at the World Wide Data Vault Consortium European conference in Hannover Germany.  I presented an overview of Disciplined Agile, some of the challenges that organizations are experiencing in their agile transformations, and how their teams can improve their way of working (WoW) via Guided Continuous Improvement (GCI).

Although all of the presentations were great, I was particularly enthralled with Bill Inmon’s keynote on Thursday.  As you may know Bill is the father of data warehousing and has written 59 (59!) books over his career, clearly putting me to shame.  Bill shared some of his experiences in extracting information from text-based sources and described several stories doing so.  One story focused on how his team had combined information culled from multiple text-based sources that together indicated that BP had a potential maintenance risk in their Gulf of Mexico operations.  Sadly his warning was ignored and several months later BP had a catastrophic oil rig failure on Deepwater Horizon.  Another story described how his team processed 5000 online postings from Nike customers and 5000 from Adidas customers.  Their analysis indicated that while Adidas was a “normal company,” that on the other hand Nike had quality problems with their shoes.  Although Bill contacted Nike to inform them, free of charge, of what he had discovered this advice was also ignored because Nike apparently already had consulting companies providing them with advice.  A year later Nike suffered a $2 billion market capitalization loss when Zion Williamson’s sneaker exploded in a basketball game watched by over 100 million people. Another text analysis project led him to discover that airlines are consistently not well liked by their customers, revealing that Bill doesn’t always end up with earth-shattering revelations.  Although the stories were interesting, Bill’s description of the techniques he was following and the challenges surrounding text-based data analytics were fascinating.

Data Vault 2 (DV2) is an extension to Inmon’s approach to data warehousing.  Dan Lindstedt, the creator of DV2, worked for years with Bill.  DV2 brings a lot of very practical strategies to data warehousing.  Furthermore, a few years ago DV2 adopted Disciplined Agile Delivery (DAD) for its process which was one of the reasons why I suspect I was invited to speak at the conference.

Kudos to everyone who made the conference a success.  I’m looking forward to next year.

We recently recorded a short discussion between Sunil Prashara, CEO of PMI; Dave Garrett, VP of Corporate Development and Innovation at PMI; and Mark Lines, co-creator of Disciplined Agile and now VP of Disciplined Agile at PMI.  In the audio recording, which you can find at Quick Podcast on PMI’s Disciplined Agile Acquisition, they share their thoughts about four interesting questions:

1. How does Disciplined Agile go beyond agile software development?
2. Why was PMI attracted to DA?
3. How will PMI leverage the IP within DA?
4. What attracted DA to PMI?

There’s a lot more goodness to come!

At the Agile 2019 conference in DC I facilitated a workshop with about 70 people where we explored the topic of how do you coach an agile data warehousing (DW)/business intelligence (BI) team. To do this we worked through four issues:

1. What challenges do you face?
2. What skills/knowledge does an agile DW/BI coach require?
3. What strategies should you apply to engage with a DW/BI team?
4. What are the qualities you should look for in an agile DW/BI coach?

The basic strategy was to introduce the issues to the class one at a time, then at their tables they would discuss the issue and write up to five ideas on sticky notes, then we’d share the ideas. Pictures of the flipcharts for each issue follow below. After the groups shared their ideas I then shared my thoughts with the class.

Issue #1: What Challenges Do You Face Coaching DW/BI Teams?

As you can see the class identified a lot of the classic issues that agile coaches face in general, such as trust issues, the teams being management-driven instead of self organizing, lack of agile skills within the team, cross-team dependencies, and being overwhelmed with work. Certainly there were DW/BI flavours of common problems, such as how to do vertical slices of DW/BI functionality and which lifecycle (agile, lean, CD, …) to follow. But there were also DW/BI specific issues, such as lack of access to data sources, knowing the actual data, and DW/BI architecture and design strategies. These DW/BI specific issues are where agile coaches tend to get hung up.

In my discussion of the challenges that we face when coaching agile DW/BI teams I shared my thoughts on the cultural impedance mismatch that exists between the agile and data communities. This mismatch makes it a bit more difficult to engage with data teams as opposed to application development teams. I also shared results of studies (2009, 2013,2016, 2018) around data quality challenges and practices – it is certainly common for teams to have to deal with technical debt, but data technical debt is both different in nature than code quality debt and the traditional data culture has led them down a very questionable (read dysfunctional) path regarding quality practices.

Issue #2: What Skills/Knowledge Does an Agile DW/BI Coach Require?

The second issue that we explored was what skills/knowledge does an agile DW/BI coach need. Once again the groups identified both classic agile coaching ideas as well as DW/BI specific ideas. Clearly you need coaching skills in order to coach a DW/BI team. But you also need to be knowledgeable about critical skills such as data modeling, data analysis, database testing, database refactoring, and others. You might not be an expert at these things, but you need to know of them and be able to guide the team in their adoption. You’ll also need to be able to speak intelligently about why some of the traditional strategies that they likely hold near and dear to their hearts (remember the cultural impedance mismatch) need to be abandoned for better, more effective strategies.

In my discussion I overviewed the “agile database techniques stack,” a collection of agile strategies and practices for database development. The stack is:

• Vertical slicing
• Clean data architecture
• Clean database design
• Agile data modeling
• Database refactoring
• Database regression testing
• Continuous database integration
• Database configuration management

As you can see, this list of techniques is fairly common from an agile point of view, albeit the corresponding data(base) versions of those techniques. The point is that the techniques exist that enable data professionals to work in an agile, and far more effective, manner. As a coach you will need to be aware of these strategies and be able to help your DW/BI team adopt them. And of course there are agile data management strategies that you need to be aware of as well.

Issue #3: What Strategies Should You Use To Engage Successfully With An Agile DW/BI Team?

The groups identified a collection of great strategies for engaging with DW/BI teams. Once again there were a lot of standard coaching strategies, a DW/BI team is still a group of people after all, but there was also a focus on strategies to address the DW/BI challenges identified earlier.

The discussion that followed the sharing of the stickies a very interesting point was brought up. I had earlier stated that my experience with coaching DW/BI teams was that it was different than coaching other types of teams, mostly because of the cultural impedance mismatch. A handful of agile DW/BI coaches in the audience disagreed with that, pointing out that the critical issue was gaining the trust and respect of the team at the start. This is true of any team, and certainly true of DW/BI teams. To do this you need to understand and appreciate the issues that they deal with and be able to show that you know how to guide them through addressing their issues. You might not be an expert in the techniques of the agile database technique stack, or other important agile data techniques, but you do know of them and can help the team learn them. So yes, engaging with an agile DW/BI team is no different on the surface, but it does require the coach to have different skills and knowledge than what your typical agile coach has.

Issues #4: What Are The Qualities You Should Look For In An Agile DW/BI Coach?

For this exercise I pretty much asked the groups to put together a list of qualities for a job ad for an Agile DW/BI coach. This is what they came up with.

Our Learnings

Here are our main learnings regarding coaching an agile DW/BI team:

• You need fundamental agile coaching skills
• You need to at least be aware of, but better yet experienced with, agile data and agile database development strategies
• To effectively engage with a DW/BI team, you need to gain their trust and respect – to do that you need to show that you understand the challenges that they face and can help them to address them
• To do so can be more challenging than with an application team due to the cultural impedance mismatch, the nature of data-oriented technical debt, and the lack of modern development skills within the data community – all of these challenges point to a greater than normal need for coaching on such teams