Disciplined Agile

Since 2001 agilists have been focused on finding ways to improve how software is developed.  This article describes what we’ve come to call the Race Car Metaphor for process improvement.

Agilists Know How to Build Great Race-Car Engines (Development Teams)

We’ve adopted fundamental strategies such as regular coordination meetings, regular demos, product owners, developer regression testing, retrospectives, and incremental releases of working software.  Disciplined teams have adopted more advanced strategies such as active stakeholder participation, continuous integration (CI), test-driven development, continuous documentation, continuous deployment, measured improvement, and incremental releases of consumable solutions (to name a few).  We experiment with new techniques to learn what works for us in the situation that we face, improving our approach as we do so in an incremental kaizen-style manner over time.  In effect we are finding ways to tune our “development engines” so that we can deliver more valuable functionality, to reduce our cycle time, and to be more productive as a team.  This is very similar to a Formula One team who over the years improves their racing car engine to deliver more power and more speed for less fuel consumption so as to help them win the race.  How to approach agile solution delivery in a streamlined manner is the focus of Disciplined Agile Delivery (DAD).

But Then We Plunk Our Great Engines Into Our Organizational Tractor

We see too many agile teams, who on their own are doing a great job at improving the way that they work, get bogged down by the organizational environment in which they work.  This is particularly true in established enterprises that have been in operation for decades and sometimes even centuries.  Your development team tries to work in an agile manner but they’re forced to conform to your PMO’s existing traditional governance strategy that requires creation of inappropriate artifacts following a lifecycle that is far from agile; they’re forced to work with a bureaucratic data management team who often takes weeks or months to do work that should be accomplished in minutes or hours; or they’re forced to work with a review-focused enterprise architecture team who hasn’t been actively involved with software development for years (to name but a few common challenges).  This is akin to putting their racing car engine into an organizational tractor.  Is it any wonder you’re not winning the race?

But We Really Need a Race Car (Disciplined DevOps)

But agile software development alone isn’t sufficient. We see too many agile teams, who on their own are doing a great job at improving the way that they work, get bogged down by their organizational environment. This is particularly true in established enterprises that have been in operation for decades and sometimes even centuries. The software developers are agile, but you still have a “quality assurance (QA)” group that insists on manual testing based on a detailed requirements document. Or it takes days and sometimes weeks to release a new version of a system into production because of your existing release management practices. You’ve got this great agile team, a great car engine, but you’ve put it into an organizational tractor. Is it any wonder you’re not seeing the desired improvements? Many organizations have come to realize that agile software development alone isn’t enough and now are focusing on DevOps. They’ve increased the scope of their improvement efforts because they realize that their race car engines really need to be put into a race car.  Enterprise-class DevOps is the focus of Disciplined DevOps.

We Need a Great Team to Run the Car (Value Streams)

But this isn’t sufficient either. Just like a race car needs a driver and pit crew to operate it, your DevOps strategy is part of your value stream. If your product management approach is based on traditional thinking that requires teams to large, detailed requirement specifications then that’s the equivalent of a pit crew putting square wheels onto the car and then holding the driver accountable for losing the race. Similarly your DevOps efforts will be for naught without the support of business operations capable of supporting customers working with updated offerings. Just like your race car requires an effective team to run it in a race, your DevOps strategy requires a supporting value stream to be effective, the focus of Disciplined Agile's Value Streams layer.

And We Need a Race Where We Can Make Money (a Disciplined Agile Enterprise)

But this still isn’t enough. A race car and team is of little value if there’s nowhere to race! They are part of a larger racing business which has multiple value streams through which they generate revenue: They make money from ticket sales, from advertisers, from merchandising, and from many other sources. Similarly, your value stream are supported by your larger organization, involved with and supporting many value streams from which you make money. This is what the Disciplined Agile Enterprise (DAE) is all about.

No Team is an Island Unto Itself

So, to summarize, an engine is part of a race car that is evolved and operated by a team of people and this race car team is part of the overall racing business. Similarly, our software/solution delivery teams are part of an overall DevOps effort, which in turn is part of your IT strategy. Your IT strategy is one aspect of your overall organizational strategy. All of this must work together smoothly, given the challenges described earlier, in order for you to have a truly agile organization. And, on top of that, you need to accomplish this given the myriad of internal and external challenges that you face. How hard could that be?

What About Non-Software Teams?

No metaphor is perfect.  This metaphor works well for software teams because the way that they work are described by the Disciplined Agile Delivery (DAD) process blade, which is part of the Disciplined DevOps layer, which in turn is part of the Value Streams layer, which is part of the overall Disciplined Agile Enterprise (DAE) layer.  But consider an agile marketing team.  It's already part of the Value Stream layer and supports Disciplined DevOps efforts as appropriate, but isn't explicitly part of the Disciplined DevOps layer.  Bottom line is that we need a second metaphor to address the realities faced by non-software teams.

In the past few years more and more teams have started to adopt agile approaches to data warehousing (DW) and business intelligence (BI).  We have been at the forefront of this movement, having developed foundational techniques around agile data modelling, database refactoring, agile database testing, and many more  that enable teams to be more agile in their approach to DW/BI development.

In this blog posting we overview how a disciplined agile DW team works in practice. The following diagram summarizes the primary and secondary development activities, by phase, that are potentially performed by the team. Primary activities are ones that add direct value to the development effort. Secondary activities tend to focus on long-term documentation that may add value in the future, but the value proposition tends to be dubious in practice so you want to be very careful in how much effort you invest in them. A good approach is to think of these as sideline activities that I would only do if the team has time to spare from primary activities.

There are several key ways in which a disciplined agile strategy for DW/BI development differs from a traditional one:

• Initial modelling and planning are high-level, not detailed.  This enables disciplined agile  teams to get the value out of modelling and planning which is to think things through without taking on the risks of over documentation and capturing details far too early.
• Development proceeds in small, incremental steps in which vertical slices of the solution are developed.  Disciplined agile teams work in priority order, thereby producing higher ROI, creating a consumable solution all the way through.  This provides greater visibility to stakeholders and provides opportunities for them to give concrete feedback earlier in the lifecycle, enabling the team to home in on what their stakeholders actually need (thereby resulting in greater stakeholder satisfaction).
• Development is usage driven, not data driven.  The goal is to produce a solution that meets the actual needs of your stakeholders, and the best way to do that is to understand how they intend to use the DW/BI solution to make better business decisions.  In practice, the structure of the data is a secondary issue at best.
• Automated regression testing occurs throughout the lifecycle across the entire architecture of the solution.  This enables the team to safely evolve their DW/BI solution incrementally, all the while being assured that their solution fulfills the requirements as they’ve been told them to date.
• The first release of the DW/BI solution will likely be developed following the Scrum-based agile lifecycle, future releases following a Continuous Delivery lifecycle.  Different situations require different approaches, and the DA toolkit purposely supports several delivery lifecycles too support a pragmatic fit-for-purpose approach.
• Stakeholders are engaged throughout the lifecycle, and changes in requirements are wholeheartedly welcomed by the team.  A changed requirement late in the lifecycle is a competitive advantage, as long as you can act on it.

For a more detailed discussion, we suggest you consider the article Disciplined Agile Data Warehousing.

In the past few years more and more teams have started to adopt agile approaches to data warehousing (DW) and business intelligence (BI).  We have been at the forefront of this movement, having developed foundational techniques around agile data modelling, database refactoring, agile database testing, and many more  that enable teams to be more agile in their approach to DW/BI development.  This blog posting overviews the differences between how a traditional team and a disciplined agile team team approaches DW/BI development.  To understand these differences, the following diagram compares the typical level of expended effort creating artifacts on traditional and disciplined agile teams.

There are several interesting differences to between the approaches. Disciplined agile DW teams will:

1. Create a high-level conceptual model early. A high-level conceptual model, or more accurately diagam at this point, identifies the critical business entity types and the relationships between them. This provides vital insight into the domain while helping to capture key domain terminology, thus helping to drive consistency of wording in other artifacts (such as user stories and epics). Traditional teams will often make the mistake of over documenting the conceptual model early in the lifecycle, injecting delay into the team (with the corresponding opportunity cost of doing so) as well as the risk of making important decisions when you and your stakeholders have the least knowledge of the actual end goal.
2. Evolve a light-weight logical data model (LDM) over time. If your agile DW team does this at all they will keep their LDM very light weight. Traditional teams will often invest heavily in their LDMs as they believe it is a mechanism to ensure quality and consistency through specifying it in. This often proves to be wishful thinking in practice. Disciplined agile teams instead invest their efforts in creating an executable specification in the form of regression tests (more on this below).
3. Evolve a detailed physical data model (PDM) over time. Disciplined agile teams realize that a PDM, when created via a data modeling tool with full round-trip engineering (it generates schemas as well as imports existing schemas), effectively becomes the source code for the database. As the requirements evolve the team will evolve the PDM to reflect these new needs, generating schema changes as needed. They can work this way because they are able to easily refactor and regression test their database. This is different from the traditional approach where they often perform detailed modeling up front. This is motivated by the mistaken belief that production database schemas are difficult to evolve, something that agilists know not to be true.
4. Develop a comprehensive regression test suite over time. These tests address several important issues. First, they validate the work of the team, showing that their work to date fulfills the requirements as they’ve been described to the team. Second, a regression test suite enables the team to safely evolve their work. Agile developers can make a small change, rerun their regression tests, and see whether they broke something (if so, then they either rollback their change or they fix what they broke). Third, when a test is written before the corresponding database schema or database code is developed, the test effectively becomes a detailed executable specification. Sophisticated agile DW teams will capture the kinds of information that were previously captured in LDMs in executable tests, and are thus much more likely to have consistent schemas than teams that still rely on static LDMs.
5. Capture critical meta-data over time. Because the rest of your organization may not be completely agile there is often a need to continue to capture key meta data about data sources. This meta data should be kept as light as possible. If there isn’t a definite need for it then don’t capture it. If someone says “but we might need it someday” then wait until someday and invest in capturing the information at that point. Furthermore, instead of capturing meta data in a static manner (i.e. as documentation) try to identify ways to capture it as tests, or to generate it automatically from other information sources. Any documentation that you write today needs to be maintained over time, slowing you down.

This blog posting is an excerpt from the detailed article Disciplined Agile Data Warehousing.

The following diagram overviews how the Disciplined Agile (DA) toolkit enables agile delivery governance.  There are two aspects to this diagram: Strategies followed by an agile or lean delivery team that enable governance and the external workflow with people or groups outside of the delivery team that support agile delivery governance.  In this blog posting we overview how to support agile delivery governance.

The diagram also called out several important workflows with other teams or activities.  This reflects that fact that IT governance addresses a wide range of concerns, many of which affect IT delivery teams. These external teams/activities support the governance of disciplined agile delivery teams in the following manner:

• Continuous improvement. People share improvement suggestions with each other to help spread good ideas throughout your organization.
• Data management. One of the outputs of your data management efforts should be common data guidance for teams to follow as appropriate.  This guidance includes such things as data naming conventions, data source meta data, and report design conventions to name a few.
• Enterprise architecture.  Your enterprise architecture activities may produce a common technology roadmap and development guidelines for teams to follow.  A common strategy is to embed enterprise architects, often in the role of Architecture Owner, on IT delivery teams to help ensure that the team follows architectural conventions and to provide a concrete feedback mechanism to the enterprise architects.  Development guidelines may include coding conventions, user interface conventions, security conventions, and many others.
• IT governance. Your IT governance activities will often provide guidance, such as corporate policies or value statements, that are above and beyond the guidance being produced by other IT activities.  IT delivery teams will provide development intelligence (DI) to the governing body to enable them to make better decisions and thereby provide better guidance to teams.
• People management. Your people management efforts, sometimes called human resource (HR) management efforts, will provide people-oriented guidance to teams.  This guidance often addressed topics such as training policies, vacation policies, roles and responsibilities, and many other things.
• Portfolio management. Your organization’s portfolio management efforts provide the initial team funding and vision to get them going, effectively giving them initial direction.  IT delivery teams provide development intelligence (DI) to the portfolio manager(s), particularly around planning and spending issues, enabling them to make better decisions.
• Product management. Your product management activities will result in the development of a business roadmap that will help to guide IT delivery teams, particularly the Product Owners on those teams.  Product management will also provide business priorities which drive the actions of your IT delivery teams.
• Release management. Your release management team will produce release guidelines and advice for IT delivery teams which helps to drive their approach to deployment.
• Stakeholders. A team’s stakeholders, but business and otherwise, will provide direction to IT delivery teams regarding what functionality they want and the priorities thereof.  Stakeholders will also provide regular feedback and other forms of input to the teams.  Delivery teams produce consumable solutions for stakeholder, updates to their plans, and development intelligence (DI) on a regular basis – this provides stakeholders with greater transparency and opportunities to steer the teams.

The bottom line is that you are being governed and we believe that you deserve to be governed well.  For more on this topic, we suggest reading the article Governing Agile Teams.

The following diagram overviews how the Disciplined Agile (DA) toolkit enables agile delivery governance.  There are two aspects to this diagram: Strategies followed by an agile or lean delivery team that enable governance and the external workflow with people or groups outside of the delivery team that support agile delivery governance.  In this blog posting we overview the strategies.

The diagram above called out several agile strategies that support the governance of disciplined agile delivery teams. These strategies are:

• Enterprise awareness. Enterprise awareness refers to the concept that disciplined agile teams realize that they work within your organization’s enterprise ecosystem, as do all other teams.  There are often existing systems in production that should not be negatively impacted by the release of the solution they are working on.  Better yet their solution will leverage existing functionality and data available in production instead of reinventing the wheel.  They will work with other teams that are working in parallel to them, striving to leverage each others work.  They will work towards your organization’s business and technical visions.  Enterprise awareness is the underpinning of effective governance.
• Release planning. Early in a project, or when a product team is first initiated, a disciplined agile team will invest some time in high-level release planning.  They do this to identify and think through any dependencies on other teams and often to identify a reasonable cost and time estimate for the current release that they are working on.  They will keep this high-level plan up-to-date as development progresses, sharing it with their stakeholders.  Release planning enables the team to answer critical governance questions regarding projected schedule and cost.
• Team dashboard.  The basic idea is that the tools used by your team should be instrumented to record important events when they occur. For example, whenever a build is run your build tool could record basic information such the date and time it was initiated, the time it took, the number of tests run, the number of successful tests, and so on. Your team management tool could record when a work item is defined, when work begins on it, when the work is validated (if appropriate), and when it is marked done. This sort of information can be recorded in a data warehouse and later reported on using business intelligence (BI) tooling via a project or portfolio dashboard. In the first DAD book we referred to this strategy as development intelligence (DI) and since the book was published this strategy has become very common within agile teams.   The real-time, accurate information radiated by a team dashboard enables the team to make better decisions and provides better transparency to stakeholders (including governance people).
• Information radiators. An information radiator is a visible display that shows something of interest to a team or their stakeholders.  Examples include a whiteboard with an architecture sketch on it, a corkboard with index cards tacked to it, and a wall-mounted monitor showing the team’s dashboard.  Information radiators enable better governance by increasing transparency.
• Active stakeholder participation. Active stakeholder participation is the practice of having on-site access to stakeholders, or at least their proxies (i.e. Product Owners).  Active stakeholders have the authority and ability to provide information and make timely decisions regarding the prioritization and scope of requirements.  This enables more effective governance through improving the team’s access to decision makers.
• Demos. On a regular basis, typically at the end of each iteration for teams following the Basic/Scrum-based lifecycle, the team either demonstrates the solution to key stakeholders. The team shows completed work and invites feedback. This practice is also called stakeholder demonstration or sprint demonstration.  This enables effective governance by increasing transparency and providing better opportunities for stakeholders to steer the team.
• Coordination meetings. The team meets for a few minutes, typically at the beginning of each day, to coordinate their activities.  The team lead facilitates the meeting and is responsible for keeping it short and focused. This practice is often called a scrum meeting or daily stand up meeting.  This enables tactical governance within the team itself through increasing internal transparency and reducing the feedback cycle within the team.
• Light-weight, risk-based milestones. Effective milestone reviews are as simple and short as possible. For a small co-located project teams milestone reviews could be as simple as a few people from the governing body, or their agents, to visit the team room and have the team spend an hour walking them through whatever is to be reviewed. For larger efforts this could be upwards to half a day and be held in meeting room. Teams in regulatory environments may need to invest a bit more effort, particularly around creation and baselining of artifacts to be reviewed and recording of action items from the review. With adoption of common agile practices such as stakeholder demos and team dashboards, described earlier, there will be less need for status discussions in milestone reviews.  The milestones suggested by the DA toolkit are risk-based, not document based.  For example, the Proven Architecture milestone is best fulfilled through development of beginning-to-end functionality that implements high-risk requirements, not the creation and review of an architecture model.  Light-weight, risk-based milestones
• Retrospectives.  A retrospective is a facilitated reflection meeting performed by the team, the goal of which is to identify potential areas of improvement. Retrospectives often last thirty to sixty minutes.  Retrospectives help teams to be more self aware and improvement focused, supporting your overall governance goal of continuous improvement.

These strategies support a light weight approach to governance while improving the overall effectiveness of the team.  Where traditional governance was something to be dreaded, agile governance should be something to be welcomed.  The bottom line is that you are being governed and we believe that you deserve to be governed well.  For more on this topic, we suggest reading the article Governing Agile Teams.