Disciplined Agile

In the past few years more and more teams have started to adopt agile approaches to data warehousing (DW) and business intelligence (BI).  We have been at the forefront of this movement, having developed foundational techniques around agile data modelling, database refactoring, agile database testing, and many more  that enable teams to be more agile in their approach to DW/BI development.

In this blog posting we overview how a disciplined agile DW team works in practice. The following diagram summarizes the primary and secondary development activities, by phase, that are potentially performed by the team. Primary activities are ones that add direct value to the development effort. Secondary activities tend to focus on long-term documentation that may add value in the future, but the value proposition tends to be dubious in practice so you want to be very careful in how much effort you invest in them. A good approach is to think of these as sideline activities that I would only do if the team has time to spare from primary activities.

There are several key ways in which a disciplined agile strategy for DW/BI development differs from a traditional one:

• Initial modelling and planning are high-level, not detailed.  This enables disciplined agile  teams to get the value out of modelling and planning which is to think things through without taking on the risks of over documentation and capturing details far too early.
• Development proceeds in small, incremental steps in which vertical slices of the solution are developed.  Disciplined agile teams work in priority order, thereby producing higher ROI, creating a consumable solution all the way through.  This provides greater visibility to stakeholders and provides opportunities for them to give concrete feedback earlier in the lifecycle, enabling the team to home in on what their stakeholders actually need (thereby resulting in greater stakeholder satisfaction).
• Development is usage driven, not data driven.  The goal is to produce a solution that meets the actual needs of your stakeholders, and the best way to do that is to understand how they intend to use the DW/BI solution to make better business decisions.  In practice, the structure of the data is a secondary issue at best.
• Automated regression testing occurs throughout the lifecycle across the entire architecture of the solution.  This enables the team to safely evolve their DW/BI solution incrementally, all the while being assured that their solution fulfills the requirements as they’ve been told them to date.
• The first release of the DW/BI solution will likely be developed following the Scrum-based agile lifecycle, future releases following a Continuous Delivery lifecycle.  Different situations require different approaches, and the DA toolkit purposely supports several delivery lifecycles too support a pragmatic fit-for-purpose approach.
• Stakeholders are engaged throughout the lifecycle, and changes in requirements are wholeheartedly welcomed by the team.  A changed requirement late in the lifecycle is a competitive advantage, as long as you can act on it.

For a more detailed discussion, we suggest you consider the article Disciplined Agile Data Warehousing.

In the past few years more and more teams have started to adopt agile approaches to data warehousing (DW) and business intelligence (BI).  We have been at the forefront of this movement, having developed foundational techniques around agile data modelling, database refactoring, agile database testing, and many more  that enable teams to be more agile in their approach to DW/BI development.  This blog posting overviews the differences between how a traditional team and a disciplined agile team team approaches DW/BI development.  To understand these differences, the following diagram compares the typical level of expended effort creating artifacts on traditional and disciplined agile teams.

There are several interesting differences to between the approaches. Disciplined agile DW teams will:

1. Create a high-level conceptual model early. A high-level conceptual model, or more accurately diagam at this point, identifies the critical business entity types and the relationships between them. This provides vital insight into the domain while helping to capture key domain terminology, thus helping to drive consistency of wording in other artifacts (such as user stories and epics). Traditional teams will often make the mistake of over documenting the conceptual model early in the lifecycle, injecting delay into the team (with the corresponding opportunity cost of doing so) as well as the risk of making important decisions when you and your stakeholders have the least knowledge of the actual end goal.
2. Evolve a light-weight logical data model (LDM) over time. If your agile DW team does this at all they will keep their LDM very light weight. Traditional teams will often invest heavily in their LDMs as they believe it is a mechanism to ensure quality and consistency through specifying it in. This often proves to be wishful thinking in practice. Disciplined agile teams instead invest their efforts in creating an executable specification in the form of regression tests (more on this below).
3. Evolve a detailed physical data model (PDM) over time. Disciplined agile teams realize that a PDM, when created via a data modeling tool with full round-trip engineering (it generates schemas as well as imports existing schemas), effectively becomes the source code for the database. As the requirements evolve the team will evolve the PDM to reflect these new needs, generating schema changes as needed. They can work this way because they are able to easily refactor and regression test their database. This is different from the traditional approach where they often perform detailed modeling up front. This is motivated by the mistaken belief that production database schemas are difficult to evolve, something that agilists know not to be true.
4. Develop a comprehensive regression test suite over time. These tests address several important issues. First, they validate the work of the team, showing that their work to date fulfills the requirements as they’ve been described to the team. Second, a regression test suite enables the team to safely evolve their work. Agile developers can make a small change, rerun their regression tests, and see whether they broke something (if so, then they either rollback their change or they fix what they broke). Third, when a test is written before the corresponding database schema or database code is developed, the test effectively becomes a detailed executable specification. Sophisticated agile DW teams will capture the kinds of information that were previously captured in LDMs in executable tests, and are thus much more likely to have consistent schemas than teams that still rely on static LDMs.
5. Capture critical meta-data over time. Because the rest of your organization may not be completely agile there is often a need to continue to capture key meta data about data sources. This meta data should be kept as light as possible. If there isn’t a definite need for it then don’t capture it. If someone says “but we might need it someday” then wait until someday and invest in capturing the information at that point. Furthermore, instead of capturing meta data in a static manner (i.e. as documentation) try to identify ways to capture it as tests, or to generate it automatically from other information sources. Any documentation that you write today needs to be maintained over time, slowing you down.

This blog posting is an excerpt from the detailed article Disciplined Agile Data Warehousing.

The following diagram overviews how the Disciplined Agile (DA) toolkit enables agile delivery governance.  There are two aspects to this diagram: Strategies followed by an agile or lean delivery team that enable governance and the external workflow with people or groups outside of the delivery team that support agile delivery governance.  In this blog posting we overview how to support agile delivery governance.

The diagram also called out several important workflows with other teams or activities.  This reflects that fact that IT governance addresses a wide range of concerns, many of which affect IT delivery teams. These external teams/activities support the governance of disciplined agile delivery teams in the following manner:

• Continuous improvement. People share improvement suggestions with each other to help spread good ideas throughout your organization.
• Data management. One of the outputs of your data management efforts should be common data guidance for teams to follow as appropriate.  This guidance includes such things as data naming conventions, data source meta data, and report design conventions to name a few.
• Enterprise architecture.  Your enterprise architecture activities may produce a common technology roadmap and development guidelines for teams to follow.  A common strategy is to embed enterprise architects, often in the role of Architecture Owner, on IT delivery teams to help ensure that the team follows architectural conventions and to provide a concrete feedback mechanism to the enterprise architects.  Development guidelines may include coding conventions, user interface conventions, security conventions, and many others.
• IT governance. Your IT governance activities will often provide guidance, such as corporate policies or value statements, that are above and beyond the guidance being produced by other IT activities.  IT delivery teams will provide development intelligence (DI) to the governing body to enable them to make better decisions and thereby provide better guidance to teams.
• People management. Your people management efforts, sometimes called human resource (HR) management efforts, will provide people-oriented guidance to teams.  This guidance often addressed topics such as training policies, vacation policies, roles and responsibilities, and many other things.
• Portfolio management. Your organization’s portfolio management efforts provide the initial team funding and vision to get them going, effectively giving them initial direction.  IT delivery teams provide development intelligence (DI) to the portfolio manager(s), particularly around planning and spending issues, enabling them to make better decisions.
• Product management. Your product management activities will result in the development of a business roadmap that will help to guide IT delivery teams, particularly the Product Owners on those teams.  Product management will also provide business priorities which drive the actions of your IT delivery teams.
• Release management. Your release management team will produce release guidelines and advice for IT delivery teams which helps to drive their approach to deployment.
• Stakeholders. A team’s stakeholders, but business and otherwise, will provide direction to IT delivery teams regarding what functionality they want and the priorities thereof.  Stakeholders will also provide regular feedback and other forms of input to the teams.  Delivery teams produce consumable solutions for stakeholder, updates to their plans, and development intelligence (DI) on a regular basis – this provides stakeholders with greater transparency and opportunities to steer the teams.

The bottom line is that you are being governed and we believe that you deserve to be governed well.  For more on this topic, we suggest reading the article Governing Agile Teams.

The following diagram overviews how the Disciplined Agile (DA) toolkit enables agile delivery governance.  There are two aspects to this diagram: Strategies followed by an agile or lean delivery team that enable governance and the external workflow with people or groups outside of the delivery team that support agile delivery governance.  In this blog posting we overview the strategies.

The diagram above called out several agile strategies that support the governance of disciplined agile delivery teams. These strategies are:

• Enterprise awareness. Enterprise awareness refers to the concept that disciplined agile teams realize that they work within your organization’s enterprise ecosystem, as do all other teams.  There are often existing systems in production that should not be negatively impacted by the release of the solution they are working on.  Better yet their solution will leverage existing functionality and data available in production instead of reinventing the wheel.  They will work with other teams that are working in parallel to them, striving to leverage each others work.  They will work towards your organization’s business and technical visions.  Enterprise awareness is the underpinning of effective governance.
• Release planning. Early in a project, or when a product team is first initiated, a disciplined agile team will invest some time in high-level release planning.  They do this to identify and think through any dependencies on other teams and often to identify a reasonable cost and time estimate for the current release that they are working on.  They will keep this high-level plan up-to-date as development progresses, sharing it with their stakeholders.  Release planning enables the team to answer critical governance questions regarding projected schedule and cost.
• Team dashboard.  The basic idea is that the tools used by your team should be instrumented to record important events when they occur. For example, whenever a build is run your build tool could record basic information such the date and time it was initiated, the time it took, the number of tests run, the number of successful tests, and so on. Your team management tool could record when a work item is defined, when work begins on it, when the work is validated (if appropriate), and when it is marked done. This sort of information can be recorded in a data warehouse and later reported on using business intelligence (BI) tooling via a project or portfolio dashboard. In the first DAD book we referred to this strategy as development intelligence (DI) and since the book was published this strategy has become very common within agile teams.   The real-time, accurate information radiated by a team dashboard enables the team to make better decisions and provides better transparency to stakeholders (including governance people).
• Information radiators. An information radiator is a visible display that shows something of interest to a team or their stakeholders.  Examples include a whiteboard with an architecture sketch on it, a corkboard with index cards tacked to it, and a wall-mounted monitor showing the team’s dashboard.  Information radiators enable better governance by increasing transparency.
• Active stakeholder participation. Active stakeholder participation is the practice of having on-site access to stakeholders, or at least their proxies (i.e. Product Owners).  Active stakeholders have the authority and ability to provide information and make timely decisions regarding the prioritization and scope of requirements.  This enables more effective governance through improving the team’s access to decision makers.
• Demos. On a regular basis, typically at the end of each iteration for teams following the Basic/Scrum-based lifecycle, the team either demonstrates the solution to key stakeholders. The team shows completed work and invites feedback. This practice is also called stakeholder demonstration or sprint demonstration.  This enables effective governance by increasing transparency and providing better opportunities for stakeholders to steer the team.
• Coordination meetings. The team meets for a few minutes, typically at the beginning of each day, to coordinate their activities.  The team lead facilitates the meeting and is responsible for keeping it short and focused. This practice is often called a scrum meeting or daily stand up meeting.  This enables tactical governance within the team itself through increasing internal transparency and reducing the feedback cycle within the team.
• Light-weight, risk-based milestones. Effective milestone reviews are as simple and short as possible. For a small co-located project teams milestone reviews could be as simple as a few people from the governing body, or their agents, to visit the team room and have the team spend an hour walking them through whatever is to be reviewed. For larger efforts this could be upwards to half a day and be held in meeting room. Teams in regulatory environments may need to invest a bit more effort, particularly around creation and baselining of artifacts to be reviewed and recording of action items from the review. With adoption of common agile practices such as stakeholder demos and team dashboards, described earlier, there will be less need for status discussions in milestone reviews.  The milestones suggested by the DA toolkit are risk-based, not document based.  For example, the Proven Architecture milestone is best fulfilled through development of beginning-to-end functionality that implements high-risk requirements, not the creation and review of an architecture model.  Light-weight, risk-based milestones
• Retrospectives.  A retrospective is a facilitated reflection meeting performed by the team, the goal of which is to identify potential areas of improvement. Retrospectives often last thirty to sixty minutes.  Retrospectives help teams to be more self aware and improvement focused, supporting your overall governance goal of continuous improvement.

These strategies support a light weight approach to governance while improving the overall effectiveness of the team.  Where traditional governance was something to be dreaded, agile governance should be something to be welcomed.  The bottom line is that you are being governed and we believe that you deserve to be governed well.  For more on this topic, we suggest reading the article Governing Agile Teams.

An important, yet seldom discussed, topic is how do you govern agile software development teams?  This is rather strange considering that agile teams are in fact being governed whether you choose to recognize this or not.  If someone is keeping an eye on the budget, or the level of quality being produced, or whether you are building something of value for your stakeholders then you are being governed.  If there are defined roles and responsibilities for team members then you’re being governed.  If you are following common programming or database guidelines, or working towards a common technical or business roadmap, or inviting outsiders to attend your coordination meetings then you are being governed.  In this blog posting we explore what is, and what isn’t, agile delivery governance

What Is Agile Delivery Governance?

Governance establishes chains of responsibil­ity, authority and communication in support of the overall enterprise’s goals and strategy. It also establishes measurements, policies, standards and control mechanisms to enable people to carry out their roles and responsibilities effectively. You do this by balancing risk versus return on investment (ROI), setting in place effective processes and practices, defining the direction and goals for a team, and defining the roles that people play within a team.

Agile delivery governance is the governance of agile delivery teams in a manner that reflects and supports the agile paradigm.  We have identified the following principles for effective agile delivery governance:

1. Collaboration with delivery teams is more effective than trying to force them to conform.  IT professionals are intellectual workers, the type of people whom are more likely to do what you want when you work with them to do so rather than tell them to do so.  For example, if you want an agile delivery team to work towards a common technical strategy (perhaps captured by your organization’s technical roadmap) you are better off having people knowledgeable with that strategy to work directly with the team rather than forcing them to fill out specific documents that will be reviewed at some point.  In this case the toolkit suggests the specific role of Architecture Owner who is responsible for this.
2. Enabling teams to do the “right thing” is more effective than trying to inspect it in. Agile team members are human, and being human their natural tendency is to do the easiest thing possible.  The implication is that for things that you want to have happen you should enable the teams to do those things, to make them easy to do.  For example, say your organization wants developers to follow common coding conventions.  One way to do this would be to hold code inspections, a time-consuming process, that validates whether the coding conventions are being followed.  An easier approach would be to adopt a code analysis tool such as CheckStyle and include it in your continuous integration (CI) strategy.  This automated approach requires no extra effort on the part of the developer and provides immediate feedback as to the quality of their code, providing them with a learning opportunity to help them improve their skills.
3. Continuous monitoring provides more timely insight than quality gate reviews.  Team dashboards that use business intelligence (BI) technology to display real-time measures generated by the use of your development tools have become very common in the past few years.  This enables both the team and their stakeholders to monitor the team’s progress in a continuous real-time manner.  This is orders of magnitude more effective than traditional “quality gate” reviews of artifacts because the information displayed on the dashboards is automatically generated as a side effect of tool usage and therefore incredibly difficult to fake, whereas team artifacts (status reports, specifications, plans, and so on) are hand-crafted and thus contain whatever information the creator(s) decide to capture.  Furthermore, after the initial cost of the dashboard technology this approach is effectively free.  In the original DAD book we referred to this strategy as development intelligence (DI).
4. Transparency into teams is provides better insight than status reports.  Through application of strategies such as information radiators, team dashboards and active stakeholder participation, the work of a disciplined agile delivery team is effectively transparent.  All of these strategies are described below.  As a result your stakeholders can discover what the current status of your team is simply by looking whenever they want, they don’t need to rely on status reports crafted by the team that may be little more than works of fiction.

What Isn’t Agile Delivery Governance?

Governance often has a bad name with developers, typically because they have had ineffective traditional strategies inflicted upon them for years.  These traditional strategies often prove to be little more than a layer of additional bureaucracy that offers little or no value to your organization.  To put your mind at ease on this issue, we’d like to point out that agile delivery governance isn’t:

• Documentation driven.  The greater levels of collaboration promoted by agile, in combination with the strategy of team dashboards alleviates the need for many of the artifacts that traditional governance strategies dictate.  Furthermore, as you will soon see, the milestones suggested by the DA toolkit are risk-driven, not documentation driven.
• Onerous.  An onerous governance strategy will not be followed by delivery teams, at best they will do just enough work to make it appear that they are following the governance process, which implies that the process has utterly failed.  As a result we’ve kept the governance strategies as light-weight as we possibly can, and when you do it right effective governance actually improves the productivity of delivery teams instead of detracting from it.  These light-weight strategies are described below.
• Management.  Governance and management are two different things. Governance looks at a team from the outside, treating it as a system that needs to have the appropriate structure and processes in place to provide a stream of value. Management, on the other hand, occurs inside the team and ensures that the structure and processes are implemented effectively.
• Optional.  Your teams are being governed, like it or not. Our philosophy is that you deserve to be governed well, which is why we’ve taken the time to describe how to do so.

You are being governed.  We believe that you deserve to be governed well.  For more on this topic, we suggest reading the article Governing Agile Teams.