Disciplined Agile

On Tuesday, February 26th I ran a webcast entitled Disciplined Agile Data Warehousing/Business Intelligence.  During the webcast we received several very good questions, some of which we had time for and some of which we didn’t get to.  Regardless, we’ve decided to answer all of them here in this blog.  In a few cases we’ve had to reword the questions to correct spelling or grammar mistakes and in a few cases we combined questions because they were effectively the same.  We have organized the questions and answers into the following categories:

1. Start here
2. Vertical slides of DW/BI functionality
3. Architecture and design
4. Refactoring databases
5. Other development practices
6. Miscellaneous
7. Where can we learn more?

Start Here

Where can we download the slides?

1. A PDF of the slide deck can be found on Slideshare.net.
2. The recording of the presentation can be found on the DAC webinars page.
3. We have also created a Disciplined Agile DW/BI poster that you can download from the DAC posters page.
   

Vertical Slices of DW/BI Functionality

How do we start with high-risk, vertical slices?

Disciplined Agile teams take a risk-value approach to prioritizing their work, an extension to Scrum’s value-driven approach.  The basic idea is that disciplined agile teams will implement the highest-risk requirements first so as to prove the architecture with working code early in the lifecycle.  This strategy works quite well for DW/BI solutions, just like any other type of solution.  To do so, you need to understand the risks that your team faces.  For a DW/BI solution, these risks may include:

• Can you access the data in key operational data sources?
• Can you process the volume of incoming data?
• Can you address data quality issues found in operational data sources?

To address these risks, look for high-value requirements whose implementation would force your to address the risks.  Implement those requirements first.

How can we include all of the stages of DW/BI development into iterations/sprints e.g. Data modeling, staging, profiling, dd, DQ, ETL, reporting, testing into single iteration?

This can be a struggle for any team new to agile, not just a DW/BI team.  The challenge is that the majority of organizations have taken a Tayloristic approach to organizing the way that they work.  They have specialists who each do a portion of the work, handing off their portion to the next person once they’ve completed it.  It is virtually impossible for specialists to get all of the work done to develop a working, vertical slice of the solution within the timeframe of a two-week iteration, let alone one that is shorter.  The overhead of specialists trying to work in a Tayloristic, “software factory” strategy is just too great.  Unfortunately the culture within the data community tends to motivate over specialization and the overhead surrounding it.

What agile teams need are generalizing specialists, T-skilled cross-functional people who work together collaboratively.  Each person has one or more specialties, they need to be able to do something useful, but they also have a general knowledge of the rest of the process and are willing to pick up new skills from one another.  When your team is made up of people like this the wait time between tasks (modelling, development, testing, …) starts to disappear as does all the bureaucracy (reviews, traceability matrices, …) around coordinating such activities.

The fundamental challenge is that you likely don’t have generalizing specialists right now.  As we like to say, you go to war with the army that you’ve got.  You need to build a team of specialists right now because that’s the type of people you have.  Insist that they produce a vertical slice of the solution during the current iteration.  There will very likely be a lot of complaining about this at first, often because the team can’t imagine how they can pull this off.  If possible, colocate them in an agile team room (sometimes called a tiger team room or war room) to get them working side-by-side.  This will help to improve communication between the people involved and provide better ways to collaborate (such as agile modelling at a whiteboard).  Push the idea that they should be doing non-solo work – such as pair programming, mob programming, or modelling with others – so as to share skills and get the work done quickly.  They will need a strong agile coach to help them to learn these new strategies and to break themselves of their ineffective specialist habits.  An important thing to observe is that many other teams have discovered how to work this way – you can too.

Vertical slice is good but if only half of a report is created in a single iteration it may not be useful to the stakeholders.  What should we do?

The idea is to get something that works completely done each iteration.  The stakeholders, often via the Product Owner (PO), will determine whether what you’ve built can be deployed into production.  This is why we use the term “potentially consumable solution“, an improvement over Scrum’s “potentially shippable software” – it’s potentially consumable, but that doesn’t mean it has to be deployed only that the option to deploy is there.

Having said this, if possible find a way to complete the entire report in a single iteration.  Sometimes easier said than done.

How to handle data coming from multiple source during data modeling within a single sprint when we are expected to develop a report in end?

This is a very common occurrence.  The solution is that you only model enough for that report at the time.  Early in the lifecycle during Inception you will have done a bit of high-level modelling to explore the initial scope and to identify your technical strategy (your architecture).  These high-level visions are fleshed out during Construction each iteration via Agile Modeling practices such as just-in-time (JIT) model storming, look-ahead modelling (backlog refinement in Scrum), and even iteration modelling (an aspect of iteration/sprint planning).

Architecture and Design

But many of the companies are not using data vault?  And may companies are reluctant in using this?

The Data Vault 2 method is not required to be agile, but it is an approach that we highly suggest due to its practicality and flexibility.

While using Data Vault how can we overcome challenges with teams supporting and creating data from various companies and geographies. Are there are control risks associated?

Regardless of the architecture and design methodology that you follow, you will have risks associated with using data from multiple sources.  Those risks tend to increase with multiple geographies or multiple companies involved.  The greater the risk, the greater the importance of having a database regression test suite in place that validates your work.

Refactoring Databases

Can we take these database refactorings as technical debt user stories?

Database refactorings are small changes to the design of your database schema (which includes functionality such as triggers or stored procedures) that improves the quality of the design without changing the semantics of the schema in a practical manner.  Examples of database refactoring includes Rename Column, Introduce Cascading Delete, and Replace One-to-Many with Associative Table.  A full catalog of database refactorings can be found here.  Because database refactorings are small you should just do them as a matter of course as you work on the database, they generally aren’t large enough to justify their own work items (such as a technical debt story).  I suggest that you read the Spilled Juice Analogy.  However, if you wanted to fix a collection of refactorings up into a single technical debt story I suppose you could do this.

Most often, data sources sit in the “business” area, that cares very little about how software are works or needs to work. Isn’t this attempt to clear data at the source something that will put strain on the organization?

I suggest that you read the Spilled Juice Analogy.  Does the business want to be able to make decisions based on information that they can trust?  Does the business want to reduce their long-term IT costs?  Does the business want IT to be able to bring solutions to market quicker?  If the answer to any of these questions is yes then they need to start treating data like an asset and invest in concrete quality techniques such as database regression testing and database refactoring.

Other Development Practices

Can we use database virtualization?

Sure, why not?

Automation – How that can be done in DW/BI project?

You can:

• Write regression tests for your database.
• Adopt continuous database integration.
• Write automated deployment scripts for database changes.

How does the practice of spikes apply?

A spike is a bit of code that is used to explore or prove a concept, typically written to pay down a technical risk.  In a DW/BI solution that might be a bit of code to:

• See how a data source is accessed
• Explore a feature of your ETL tool
• Work with a BI tool for the first time
• Explore whether a data source can handle expected volume
• And many more technical risks.

How do you make sure that documentation is updated and yet consumes less time which is accommodatable within the Agile iteration/sprint of 2 weeks? Without documentation the developers who come into the project at a later point of time are in no man’s land without updated documentation?

See:

• Agile/Lean Documentation Strategies
• Practice: Document Continuously
• Practice: Document Late
• When should we create a document on an agile team?

Miscellaneous

How can enterprise data governance fit into an agile DW/BI mindset?

See:

• Lean Data Governance.
• Supporting Agile Delivery Team Governance.

What lifecycle would be most appropriate for projects implementing COTS software solutions (like ERP) within the company so investment would be maximized?

It depends on the situation that you face, including the skills of the people involved.  I would think that your best bet is the Agile/Basic lifecycle that is based on Scrum.

Where Can We Learn More?

At the DAC posters page you can download the Disciplined Agile DW/BI poster (amongst many others).

We have a detailed article entitled Disciplined Agile Data Warehousing that you will find informative.

In IT we are often asked to estimate the expected time/schedule or cost of software development. Sadly, the desire of stakeholders to have “predictable” schedules or costs results in significant dysfunction within a software development team.  When a software team is  forced by their stakeholders to commit to a schedule/cost they must then ensure that the schedule/cost doesn’t slip. For example, to protect themselves from increased time and cost due to scope creep, software development teams will make it difficult for stakeholders to change their requirements during Construction and even go so far as to drop promised scope late in a project. The desire of stakeholders to reduce their financial risk often results in behaviors by the software development team that ensure that stakeholders don’t get what they actually want.  Naturally IT gets blamed for this.

We need to do better.  In this blog we summarize the things that we know to be true about software development estimation. In no particular order, they are:

1. Estimates are guesses. Look up the word in the dictionary – An estimate is a rough approximation or calculation, in other words a guess. Unfortunately, too many people think that estimates are promises, or worse yet guarantees.  In our opinion “guesstimate” is a far more appropriate word that “estimate”.
2. Scope on IT projects is a moving target. Our stakeholders struggle to tell us what they want and even when they do they change their minds anyway.  Any guesstimate based on varying scope must also vary in kind.
3. Guesstimates are probability distributions. Although your stakeholders may ask for a fixed amount guesstimate, for example “this will cost $1 million”, the reality is that there’s a chance the cost will be less than $1 million and a very good chance that it will be more.  There is ample evidence that the initial estimate for a software development project should be given in a range of -25% to +75%, so your million dollar project should be quoted as a range of $750,000 and $1,750,000.  This is shown in the diagram above by the green distribution curve.  In many organizations this can be politically difficult to do, and strangely enough in many cases stakeholders prefer to be lied to (it’s going to be $1,000,000) rather than be told the truth.
4. Guesstimates must reflect the quality of the inputs.  A guesstimate needs to reflect the quality of the information going into it – if your scope is fuzzy your guesstimate based on that scope needs to be equally fuzzy.  Sometimes stakeholders want a guesstimate with a tight range, perhaps +/- 10% (the red curve in the diagram), early in a project.  To provide a tight range such as this you need to have a very good understanding of the requirements and the design.  Early in the software development process this can only be done through more detailed modeling, an expensive and risky proposition which often proves to be a wasted effort because the requirements will evolve over time anyway.
5. Guesstimates anchor perception. The primary danger of providing guesstimates to people is that they believe them.  Tell someone that it’s going to be $1,000,000 and they fixate on that cost even while they are changing their minds.  Tell them that it’s going to be between $750,000 and $1,750,000 and most people will fixate on the cost of $750,000.  Some people will focus on the average cost of $1,250,000 even though the median was $1,000,000 (guesstimates are in effect Weibull probability distributions).
6. It’s easier to guesstimate small things. ‘Nuff said.
7. It’s easier to guesstimate work you’re just about to do instead of work in the distant future. It is much easier to identify the details of work to be done right now, and thus turn a large piece of work into a collection of smaller pieces that are easier to guesstimate.  In part this is because you have a much better understanding of the current situation you are working in and in part because you are more focused on the here and now.
8. The people doing the work will likely give a better guesstimate.   They are more motivated to get the guesstimate right, particularly when they must commit to it, and have a much better idea of their abilities.  Granted, someone may need to coach people through the guesstimation effort.  In Disciplined Agile this is a responsibility of the team lead.
9. Someone who has done the work before will give a better guesstimate than someone who hasn’t. Experience counts.
10. Guesstimates reflect the situation that you face.  Which organizational situation do you think will result in a short schedule and lower cost: Five people co-located in a single room or the same five people working from different locations in difficult time zones?  Or how about a team working under regulatory constraints versus the same team without those constraints?  Context counts.
11. Multiple guesstimates are better than a single guesstimate. Getting guesstimates from several people provides insights from several points of view, hopefully prompting an intelligent conversation that enables you to develop a guesstimate with better confidence.  Similarly, the same person producing guesstimates for the same piece of work using different guesstimation strategies will also provide a range of answers that you can combine.
12. Guesstimates should be updated over time. As your understanding of what stakeholders want improves, and your understanding of how well your team works together, you should update your guesstimates.  As your understanding of the fundamental inputs into your estimate improves you are able to produce a better estimate, thus enabling the stakeholders of that guesstimate to make better decisions.
13. It costs money to produce a guesstimate. The precision of an estimate is driven by the detail and stability of the information going into it. Want a tighter range on your estimate?  Then you’re going to have to have a better handle on the requirements, design, and capabilities of the team doing the work.  This greater precision requires greater cost.  The fundamental question posed by the #NoEstimates community is effectively “Is the value of improved decision making capability from having the guesstimate greater than the cost of creating the guesstimate?” The implication is that you must ensure the cost is much less than the benefit, hence their focus on finding ways to streamline and even eliminate the guesstimation effort.
14. Guesstimation is far more art than science. See point #1 about estimates being guesses.  The best guesstimates are done by the people doing the work, just before they need to do the work, for small pieces of work.
15. Formal software guesstimation schemes are little more than a scientific façade. Function point counting, feature point counting, and COCOMO II are all examples of formal strategies.  They boil down to generating numeric guesses from your detailed requirements and design, plugging these guesses into an algorithm which then produces a guesstimate. These are all expensive strategies (they require detailed requirements and design work to be performed) that prove to be risky (because they often force you into a waterfall approach) in practice.  Yes, they do in fact work to some extent, but in practice there are much less expensive and less risky strategies to choose from.  People like these type of guesstimation strategies because they provide a false sense of security due to their complexity and cost.
16. Past history isn’t as valuable as people hope.  Some formal guesstimation strategies are based on past history, but this proves to be a false foundation from which to build upon for several reasons.  First, people have different levels of capability which change over time as they learn. Capers Jones has shown that developers have productivity ranges of 1 to 25, the implication being that if you don’t know exactly who is on a team and how well they work together your corporate history will be questionable.   Second, technologies evolve quickly so past history from working with older versions of technologies or completely different technologies becomes questionable at best.  Third, people and teams change (hopefully for the better) over time, implying that an input into your guesstimate is fuzzy at best.  Fourth, because every team is unique and faces a unique situation basing estimates on past history from other teams in different situations proves questionable.
17. Beware professional guesstimators. They tend to break many of the rules we’ve described above.

To summarize, when you are required to provide estimates for your software development efforts that you should take a pragmatic, light-weight approach to doing so.  This blog posting has provided many practical insights that should help guide your decisions.  These insights and many more, are built right into the Disciplined Agile (DA) toolkit.

When transitioning to an agile mindset people invariably ask about how documentation is addressed on agile teams.  Some documents, such as system overview documentation or operations manuals, are still a valuable part of the overall solution being delivered by the agile team.  More often than not a document, such as a logical data model (LDM) or detailed requirements model, isn’t needed at all because it can be replaced with a more effective strategy or the document can be greatly simplified compared with what a traditional team would develop.  This of course is a shock to most traditionalists, particularly for those who currently spend most of their time creating such documents.

To help people understand the agile approach to documentation creation, we find that it’s valuable to describe the logic that disciplined agilists follow.  This logic is captured in the following flow chart.

Let’s walk through the decision points one at a time:

1. Do you need a document?  There is a significant difference between needing a document, perhaps to persist important information, and wanting a document.  If you’re creating the document to communicate information to someone else then you really need to question its creation.  Research into media richness communication has found that detailed documentation is the least effective strategy available to you – better options include overview documentation, video conferencing, and of course face-to-face (F2F) discussion. For a second example, many organizations still follow a traditional, documentation-based approach to IT governance and as a result many teams are forced to create documents solely because someone in a governance roles wants it to be created (or at least the team perceives that they want it).  Sometimes people want a document because they fear that the information it would contain would be lost, not realizing that the older documentation becomes the less it is trusted (see Documentation CRUFT), so in effect the information is effectively lost even when it is captured as written documentation.  The point is that many documents aren’t really needed, they are merely wanted – if the latter, isn’t it better to deal with the real people and help people to recognize they don’t really need the document after all?
2. Is there a better option?  In many cases there are significantly better alternatives to writing documentation.  For example, instead of creating a detailed written requirement specification when you follow an acceptance test-driven development (ATDD), also known as a behavior driven development (BDD) approach, you capture requirements in the form of executable specifications.  Granted, this takes a different set of skills and tools to perform, but in the long run it offers far greater value to your team than written specifications.
3. Will you maintain the document? If you’re not willing to maintain a document throughout what should be its useful lifespan then don’t create it.  If this isn’t an option, then at minimum only work on it until the point where you’re still able to viably keep it up to date.  Documents that aren’t maintained aren’t trusted, and therefore not used.
4. Do you understand what the consumer of the document actually needs?  The only way that you’ll be able to create an effective document is if you know what the audience for that document actually needs, and the best way to do that is to work with them closely.  Your goal is to create a document that is just barely good enough (JBGE), or just sufficient, that fulfills their needs and no more.  For example, instead of creating a detailed architecture model using a software-based modelling tool, co-located agile teams will often prefer to create whiteboard sketches which they leave on their workroom walls.  Yes, a pretty architecture model would be nice to have.  But the whiteboard sketch gets the job done, it is much easier to evolve, and much less expensive to create.

When an organization transforms to agile many traditional IT professionals will struggle at first with taking an agile approach to documentation.  In traditional software development, and in particular traditional IT governance, documentation is used as a crutch and worse yet a band-aid over organizational dysfunction.  We can no longer afford this and must instead be smarter about our approach to whether and how we write documentation.

For more information about agile approaches to documentation, we suggest you read the article Agile/Lean Documentation: Strategies for Agile Software Development.

Since 2001 agilists have been focused on finding ways to improve how software is developed.  This article describes what we’ve come to call the Race Car Metaphor for process improvement.

Agilists Know How to Build Great Race-Car Engines (Development Teams)

We’ve adopted fundamental strategies such as regular coordination meetings, regular demos, product owners, developer regression testing, retrospectives, and incremental releases of working software.  Disciplined teams have adopted more advanced strategies such as active stakeholder participation, continuous integration (CI), test-driven development, continuous documentation, continuous deployment, measured improvement, and incremental releases of consumable solutions (to name a few).  We experiment with new techniques to learn what works for us in the situation that we face, improving our approach as we do so in an incremental kaizen-style manner over time.  In effect we are finding ways to tune our “development engines” so that we can deliver more valuable functionality, to reduce our cycle time, and to be more productive as a team.  This is very similar to a Formula One team who over the years improves their racing car engine to deliver more power and more speed for less fuel consumption so as to help them win the race.  How to approach agile solution delivery in a streamlined manner is the focus of Disciplined Agile Delivery (DAD).

But Then We Plunk Our Great Engines Into Our Organizational Tractor

We see too many agile teams, who on their own are doing a great job at improving the way that they work, get bogged down by the organizational environment in which they work.  This is particularly true in established enterprises that have been in operation for decades and sometimes even centuries.  Your development team tries to work in an agile manner but they’re forced to conform to your PMO’s existing traditional governance strategy that requires creation of inappropriate artifacts following a lifecycle that is far from agile; they’re forced to work with a bureaucratic data management team who often takes weeks or months to do work that should be accomplished in minutes or hours; or they’re forced to work with a review-focused enterprise architecture team who hasn’t been actively involved with software development for years (to name but a few common challenges).  This is akin to putting their racing car engine into an organizational tractor.  Is it any wonder you’re not winning the race?

But We Really Need a Race Car (Disciplined DevOps)

But agile software development alone isn’t sufficient. We see too many agile teams, who on their own are doing a great job at improving the way that they work, get bogged down by their organizational environment. This is particularly true in established enterprises that have been in operation for decades and sometimes even centuries. The software developers are agile, but you still have a “quality assurance (QA)” group that insists on manual testing based on a detailed requirements document. Or it takes days and sometimes weeks to release a new version of a system into production because of your existing release management practices. You’ve got this great agile team, a great car engine, but you’ve put it into an organizational tractor. Is it any wonder you’re not seeing the desired improvements? Many organizations have come to realize that agile software development alone isn’t enough and now are focusing on DevOps. They’ve increased the scope of their improvement efforts because they realize that their race car engines really need to be put into a race car.  Enterprise-class DevOps is the focus of Disciplined DevOps.

We Need a Great Team to Run the Car (Value Streams)

But this isn’t sufficient either. Just like a race car needs a driver and pit crew to operate it, your DevOps strategy is part of your value stream. If your product management approach is based on traditional thinking that requires teams to large, detailed requirement specifications then that’s the equivalent of a pit crew putting square wheels onto the car and then holding the driver accountable for losing the race. Similarly your DevOps efforts will be for naught without the support of business operations capable of supporting customers working with updated offerings. Just like your race car requires an effective team to run it in a race, your DevOps strategy requires a supporting value stream to be effective, the focus of Disciplined Agile's Value Streams layer.

And We Need a Race Where We Can Make Money (a Disciplined Agile Enterprise)

But this still isn’t enough. A race car and team is of little value if there’s nowhere to race! They are part of a larger racing business which has multiple value streams through which they generate revenue: They make money from ticket sales, from advertisers, from merchandising, and from many other sources. Similarly, your value stream are supported by your larger organization, involved with and supporting many value streams from which you make money. This is what the Disciplined Agile Enterprise (DAE) is all about.

No Team is an Island Unto Itself

So, to summarize, an engine is part of a race car that is evolved and operated by a team of people and this race car team is part of the overall racing business. Similarly, our software/solution delivery teams are part of an overall DevOps effort, which in turn is part of your IT strategy. Your IT strategy is one aspect of your overall organizational strategy. All of this must work together smoothly, given the challenges described earlier, in order for you to have a truly agile organization. And, on top of that, you need to accomplish this given the myriad of internal and external challenges that you face. How hard could that be?

What About Non-Software Teams?

No metaphor is perfect.  This metaphor works well for software teams because the way that they work are described by the Disciplined Agile Delivery (DAD) process blade, which is part of the Disciplined DevOps layer, which in turn is part of the Value Streams layer, which is part of the overall Disciplined Agile Enterprise (DAE) layer.  But consider an agile marketing team.  It's already part of the Value Stream layer and supports Disciplined DevOps efforts as appropriate, but isn't explicitly part of the Disciplined DevOps layer.  Bottom line is that we need a second metaphor to address the realities faced by non-software teams.

In the past few years more and more teams have started to adopt agile approaches to data warehousing (DW) and business intelligence (BI).  We have been at the forefront of this movement, having developed foundational techniques around agile data modelling, database refactoring, agile database testing, and many more  that enable teams to be more agile in their approach to DW/BI development.

In this blog posting we overview how a disciplined agile DW team works in practice. The following diagram summarizes the primary and secondary development activities, by phase, that are potentially performed by the team. Primary activities are ones that add direct value to the development effort. Secondary activities tend to focus on long-term documentation that may add value in the future, but the value proposition tends to be dubious in practice so you want to be very careful in how much effort you invest in them. A good approach is to think of these as sideline activities that I would only do if the team has time to spare from primary activities.

There are several key ways in which a disciplined agile strategy for DW/BI development differs from a traditional one:

• Initial modelling and planning are high-level, not detailed.  This enables disciplined agile  teams to get the value out of modelling and planning which is to think things through without taking on the risks of over documentation and capturing details far too early.
• Development proceeds in small, incremental steps in which vertical slices of the solution are developed.  Disciplined agile teams work in priority order, thereby producing higher ROI, creating a consumable solution all the way through.  This provides greater visibility to stakeholders and provides opportunities for them to give concrete feedback earlier in the lifecycle, enabling the team to home in on what their stakeholders actually need (thereby resulting in greater stakeholder satisfaction).
• Development is usage driven, not data driven.  The goal is to produce a solution that meets the actual needs of your stakeholders, and the best way to do that is to understand how they intend to use the DW/BI solution to make better business decisions.  In practice, the structure of the data is a secondary issue at best.
• Automated regression testing occurs throughout the lifecycle across the entire architecture of the solution.  This enables the team to safely evolve their DW/BI solution incrementally, all the while being assured that their solution fulfills the requirements as they’ve been told them to date.
• The first release of the DW/BI solution will likely be developed following the Scrum-based agile lifecycle, future releases following a Continuous Delivery lifecycle.  Different situations require different approaches, and the DA toolkit purposely supports several delivery lifecycles too support a pragmatic fit-for-purpose approach.
• Stakeholders are engaged throughout the lifecycle, and changes in requirements are wholeheartedly welcomed by the team.  A changed requirement late in the lifecycle is a competitive advantage, as long as you can act on it.

For a more detailed discussion, we suggest you consider the article Disciplined Agile Data Warehousing.