Disciplined Agile

In IT we are often asked to estimate the expected time/schedule or cost of software development. Sadly, the desire of stakeholders to have “predictable” schedules or costs results in significant dysfunction within a software development team.  When a software team is  forced by their stakeholders to commit to a schedule/cost they must then ensure that the schedule/cost doesn’t slip. For example, to protect themselves from increased time and cost due to scope creep, software development teams will make it difficult for stakeholders to change their requirements during Construction and even go so far as to drop promised scope late in a project. The desire of stakeholders to reduce their financial risk often results in behaviors by the software development team that ensure that stakeholders don’t get what they actually want.  Naturally IT gets blamed for this.

We need to do better.  In this blog we summarize the things that we know to be true about software development estimation. In no particular order, they are:

1. Estimates are guesses. Look up the word in the dictionary – An estimate is a rough approximation or calculation, in other words a guess. Unfortunately, too many people think that estimates are promises, or worse yet guarantees.  In our opinion “guesstimate” is a far more appropriate word that “estimate”.
2. Scope on IT projects is a moving target. Our stakeholders struggle to tell us what they want and even when they do they change their minds anyway.  Any guesstimate based on varying scope must also vary in kind.
3. Guesstimates are probability distributions. Although your stakeholders may ask for a fixed amount guesstimate, for example “this will cost $1 million”, the reality is that there’s a chance the cost will be less than $1 million and a very good chance that it will be more.  There is ample evidence that the initial estimate for a software development project should be given in a range of -25% to +75%, so your million dollar project should be quoted as a range of $750,000 and $1,750,000.  This is shown in the diagram above by the green distribution curve.  In many organizations this can be politically difficult to do, and strangely enough in many cases stakeholders prefer to be lied to (it’s going to be $1,000,000) rather than be told the truth.
4. Guesstimates must reflect the quality of the inputs.  A guesstimate needs to reflect the quality of the information going into it – if your scope is fuzzy your guesstimate based on that scope needs to be equally fuzzy.  Sometimes stakeholders want a guesstimate with a tight range, perhaps +/- 10% (the red curve in the diagram), early in a project.  To provide a tight range such as this you need to have a very good understanding of the requirements and the design.  Early in the software development process this can only be done through more detailed modeling, an expensive and risky proposition which often proves to be a wasted effort because the requirements will evolve over time anyway.
5. Guesstimates anchor perception. The primary danger of providing guesstimates to people is that they believe them.  Tell someone that it’s going to be $1,000,000 and they fixate on that cost even while they are changing their minds.  Tell them that it’s going to be between $750,000 and $1,750,000 and most people will fixate on the cost of $750,000.  Some people will focus on the average cost of $1,250,000 even though the median was $1,000,000 (guesstimates are in effect Weibull probability distributions).
6. It’s easier to guesstimate small things. ‘Nuff said.
7. It’s easier to guesstimate work you’re just about to do instead of work in the distant future. It is much easier to identify the details of work to be done right now, and thus turn a large piece of work into a collection of smaller pieces that are easier to guesstimate.  In part this is because you have a much better understanding of the current situation you are working in and in part because you are more focused on the here and now.
8. The people doing the work will likely give a better guesstimate.   They are more motivated to get the guesstimate right, particularly when they must commit to it, and have a much better idea of their abilities.  Granted, someone may need to coach people through the guesstimation effort.  In Disciplined Agile this is a responsibility of the team lead.
9. Someone who has done the work before will give a better guesstimate than someone who hasn’t. Experience counts.
10. Guesstimates reflect the situation that you face.  Which organizational situation do you think will result in a short schedule and lower cost: Five people co-located in a single room or the same five people working from different locations in difficult time zones?  Or how about a team working under regulatory constraints versus the same team without those constraints?  Context counts.
11. Multiple guesstimates are better than a single guesstimate. Getting guesstimates from several people provides insights from several points of view, hopefully prompting an intelligent conversation that enables you to develop a guesstimate with better confidence.  Similarly, the same person producing guesstimates for the same piece of work using different guesstimation strategies will also provide a range of answers that you can combine.
12. Guesstimates should be updated over time. As your understanding of what stakeholders want improves, and your understanding of how well your team works together, you should update your guesstimates.  As your understanding of the fundamental inputs into your estimate improves you are able to produce a better estimate, thus enabling the stakeholders of that guesstimate to make better decisions.
13. It costs money to produce a guesstimate. The precision of an estimate is driven by the detail and stability of the information going into it. Want a tighter range on your estimate?  Then you’re going to have to have a better handle on the requirements, design, and capabilities of the team doing the work.  This greater precision requires greater cost.  The fundamental question posed by the #NoEstimates community is effectively “Is the value of improved decision making capability from having the guesstimate greater than the cost of creating the guesstimate?” The implication is that you must ensure the cost is much less than the benefit, hence their focus on finding ways to streamline and even eliminate the guesstimation effort.
14. Guesstimation is far more art than science. See point #1 about estimates being guesses.  The best guesstimates are done by the people doing the work, just before they need to do the work, for small pieces of work.
15. Formal software guesstimation schemes are little more than a scientific façade. Function point counting, feature point counting, and COCOMO II are all examples of formal strategies.  They boil down to generating numeric guesses from your detailed requirements and design, plugging these guesses into an algorithm which then produces a guesstimate. These are all expensive strategies (they require detailed requirements and design work to be performed) that prove to be risky (because they often force you into a waterfall approach) in practice.  Yes, they do in fact work to some extent, but in practice there are much less expensive and less risky strategies to choose from.  People like these type of guesstimation strategies because they provide a false sense of security due to their complexity and cost.
16. Past history isn’t as valuable as people hope.  Some formal guesstimation strategies are based on past history, but this proves to be a false foundation from which to build upon for several reasons.  First, people have different levels of capability which change over time as they learn. Capers Jones has shown that developers have productivity ranges of 1 to 25, the implication being that if you don’t know exactly who is on a team and how well they work together your corporate history will be questionable.   Second, technologies evolve quickly so past history from working with older versions of technologies or completely different technologies becomes questionable at best.  Third, people and teams change (hopefully for the better) over time, implying that an input into your guesstimate is fuzzy at best.  Fourth, because every team is unique and faces a unique situation basing estimates on past history from other teams in different situations proves questionable.
17. Beware professional guesstimators. They tend to break many of the rules we’ve described above.

To summarize, when you are required to provide estimates for your software development efforts that you should take a pragmatic, light-weight approach to doing so.  This blog posting has provided many practical insights that should help guide your decisions.  These insights and many more, are built right into the Disciplined Agile (DA) toolkit.

When transitioning to an agile mindset people invariably ask about how documentation is addressed on agile teams.  Some documents, such as system overview documentation or operations manuals, are still a valuable part of the overall solution being delivered by the agile team.  More often than not a document, such as a logical data model (LDM) or detailed requirements model, isn’t needed at all because it can be replaced with a more effective strategy or the document can be greatly simplified compared with what a traditional team would develop.  This of course is a shock to most traditionalists, particularly for those who currently spend most of their time creating such documents.

To help people understand the agile approach to documentation creation, we find that it’s valuable to describe the logic that disciplined agilists follow.  This logic is captured in the following flow chart.

Let’s walk through the decision points one at a time:

1. Do you need a document?  There is a significant difference between needing a document, perhaps to persist important information, and wanting a document.  If you’re creating the document to communicate information to someone else then you really need to question its creation.  Research into media richness communication has found that detailed documentation is the least effective strategy available to you – better options include overview documentation, video conferencing, and of course face-to-face (F2F) discussion. For a second example, many organizations still follow a traditional, documentation-based approach to IT governance and as a result many teams are forced to create documents solely because someone in a governance roles wants it to be created (or at least the team perceives that they want it).  Sometimes people want a document because they fear that the information it would contain would be lost, not realizing that the older documentation becomes the less it is trusted (see Documentation CRUFT), so in effect the information is effectively lost even when it is captured as written documentation.  The point is that many documents aren’t really needed, they are merely wanted – if the latter, isn’t it better to deal with the real people and help people to recognize they don’t really need the document after all?
2. Is there a better option?  In many cases there are significantly better alternatives to writing documentation.  For example, instead of creating a detailed written requirement specification when you follow an acceptance test-driven development (ATDD), also known as a behavior driven development (BDD) approach, you capture requirements in the form of executable specifications.  Granted, this takes a different set of skills and tools to perform, but in the long run it offers far greater value to your team than written specifications.
3. Will you maintain the document? If you’re not willing to maintain a document throughout what should be its useful lifespan then don’t create it.  If this isn’t an option, then at minimum only work on it until the point where you’re still able to viably keep it up to date.  Documents that aren’t maintained aren’t trusted, and therefore not used.
4. Do you understand what the consumer of the document actually needs?  The only way that you’ll be able to create an effective document is if you know what the audience for that document actually needs, and the best way to do that is to work with them closely.  Your goal is to create a document that is just barely good enough (JBGE), or just sufficient, that fulfills their needs and no more.  For example, instead of creating a detailed architecture model using a software-based modelling tool, co-located agile teams will often prefer to create whiteboard sketches which they leave on their workroom walls.  Yes, a pretty architecture model would be nice to have.  But the whiteboard sketch gets the job done, it is much easier to evolve, and much less expensive to create.

When an organization transforms to agile many traditional IT professionals will struggle at first with taking an agile approach to documentation.  In traditional software development, and in particular traditional IT governance, documentation is used as a crutch and worse yet a band-aid over organizational dysfunction.  We can no longer afford this and must instead be smarter about our approach to whether and how we write documentation.

For more information about agile approaches to documentation, we suggest you read the article Agile/Lean Documentation: Strategies for Agile Software Development.

Since 2001 agilists have been focused on finding ways to improve how software is developed.  This article describes what we’ve come to call the Race Car Metaphor for process improvement.

Agilists Know How to Build Great Race-Car Engines (Development Teams)

We’ve adopted fundamental strategies such as regular coordination meetings, regular demos, product owners, developer regression testing, retrospectives, and incremental releases of working software.  Disciplined teams have adopted more advanced strategies such as active stakeholder participation, continuous integration (CI), test-driven development, continuous documentation, continuous deployment, measured improvement, and incremental releases of consumable solutions (to name a few).  We experiment with new techniques to learn what works for us in the situation that we face, improving our approach as we do so in an incremental kaizen-style manner over time.  In effect we are finding ways to tune our “development engines” so that we can deliver more valuable functionality, to reduce our cycle time, and to be more productive as a team.  This is very similar to a Formula One team who over the years improves their racing car engine to deliver more power and more speed for less fuel consumption so as to help them win the race.  How to approach agile solution delivery in a streamlined manner is the focus of Disciplined Agile Delivery (DAD).

But Then We Plunk Our Great Engines Into Our Organizational Tractor

We see too many agile teams, who on their own are doing a great job at improving the way that they work, get bogged down by the organizational environment in which they work.  This is particularly true in established enterprises that have been in operation for decades and sometimes even centuries.  Your development team tries to work in an agile manner but they’re forced to conform to your PMO’s existing traditional governance strategy that requires creation of inappropriate artifacts following a lifecycle that is far from agile; they’re forced to work with a bureaucratic data management team who often takes weeks or months to do work that should be accomplished in minutes or hours; or they’re forced to work with a review-focused enterprise architecture team who hasn’t been actively involved with software development for years (to name but a few common challenges).  This is akin to putting their racing car engine into an organizational tractor.  Is it any wonder you’re not winning the race?

But We Really Need a Race Car (Disciplined DevOps)

But agile software development alone isn’t sufficient. We see too many agile teams, who on their own are doing a great job at improving the way that they work, get bogged down by their organizational environment. This is particularly true in established enterprises that have been in operation for decades and sometimes even centuries. The software developers are agile, but you still have a “quality assurance (QA)” group that insists on manual testing based on a detailed requirements document. Or it takes days and sometimes weeks to release a new version of a system into production because of your existing release management practices. You’ve got this great agile team, a great car engine, but you’ve put it into an organizational tractor. Is it any wonder you’re not seeing the desired improvements? Many organizations have come to realize that agile software development alone isn’t enough and now are focusing on DevOps. They’ve increased the scope of their improvement efforts because they realize that their race car engines really need to be put into a race car.  Enterprise-class DevOps is the focus of Disciplined DevOps.

We Need a Great Team to Run the Car (Value Streams)

But this isn’t sufficient either. Just like a race car needs a driver and pit crew to operate it, your DevOps strategy is part of your value stream. If your product management approach is based on traditional thinking that requires teams to large, detailed requirement specifications then that’s the equivalent of a pit crew putting square wheels onto the car and then holding the driver accountable for losing the race. Similarly your DevOps efforts will be for naught without the support of business operations capable of supporting customers working with updated offerings. Just like your race car requires an effective team to run it in a race, your DevOps strategy requires a supporting value stream to be effective, the focus of Disciplined Agile's Value Streams layer.

And We Need a Race Where We Can Make Money (a Disciplined Agile Enterprise)

But this still isn’t enough. A race car and team is of little value if there’s nowhere to race! They are part of a larger racing business which has multiple value streams through which they generate revenue: They make money from ticket sales, from advertisers, from merchandising, and from many other sources. Similarly, your value stream are supported by your larger organization, involved with and supporting many value streams from which you make money. This is what the Disciplined Agile Enterprise (DAE) is all about.

No Team is an Island Unto Itself

So, to summarize, an engine is part of a race car that is evolved and operated by a team of people and this race car team is part of the overall racing business. Similarly, our software/solution delivery teams are part of an overall DevOps effort, which in turn is part of your IT strategy. Your IT strategy is one aspect of your overall organizational strategy. All of this must work together smoothly, given the challenges described earlier, in order for you to have a truly agile organization. And, on top of that, you need to accomplish this given the myriad of internal and external challenges that you face. How hard could that be?

What About Non-Software Teams?

No metaphor is perfect.  This metaphor works well for software teams because the way that they work are described by the Disciplined Agile Delivery (DAD) process blade, which is part of the Disciplined DevOps layer, which in turn is part of the Value Streams layer, which is part of the overall Disciplined Agile Enterprise (DAE) layer.  But consider an agile marketing team.  It's already part of the Value Stream layer and supports Disciplined DevOps efforts as appropriate, but isn't explicitly part of the Disciplined DevOps layer.  Bottom line is that we need a second metaphor to address the realities faced by non-software teams.

In the past few years more and more teams have started to adopt agile approaches to data warehousing (DW) and business intelligence (BI).  We have been at the forefront of this movement, having developed foundational techniques around agile data modelling, database refactoring, agile database testing, and many more  that enable teams to be more agile in their approach to DW/BI development.

In this blog posting we overview how a disciplined agile DW team works in practice. The following diagram summarizes the primary and secondary development activities, by phase, that are potentially performed by the team. Primary activities are ones that add direct value to the development effort. Secondary activities tend to focus on long-term documentation that may add value in the future, but the value proposition tends to be dubious in practice so you want to be very careful in how much effort you invest in them. A good approach is to think of these as sideline activities that I would only do if the team has time to spare from primary activities.

There are several key ways in which a disciplined agile strategy for DW/BI development differs from a traditional one:

• Initial modelling and planning are high-level, not detailed.  This enables disciplined agile  teams to get the value out of modelling and planning which is to think things through without taking on the risks of over documentation and capturing details far too early.
• Development proceeds in small, incremental steps in which vertical slices of the solution are developed.  Disciplined agile teams work in priority order, thereby producing higher ROI, creating a consumable solution all the way through.  This provides greater visibility to stakeholders and provides opportunities for them to give concrete feedback earlier in the lifecycle, enabling the team to home in on what their stakeholders actually need (thereby resulting in greater stakeholder satisfaction).
• Development is usage driven, not data driven.  The goal is to produce a solution that meets the actual needs of your stakeholders, and the best way to do that is to understand how they intend to use the DW/BI solution to make better business decisions.  In practice, the structure of the data is a secondary issue at best.
• Automated regression testing occurs throughout the lifecycle across the entire architecture of the solution.  This enables the team to safely evolve their DW/BI solution incrementally, all the while being assured that their solution fulfills the requirements as they’ve been told them to date.
• The first release of the DW/BI solution will likely be developed following the Scrum-based agile lifecycle, future releases following a Continuous Delivery lifecycle.  Different situations require different approaches, and the DA toolkit purposely supports several delivery lifecycles too support a pragmatic fit-for-purpose approach.
• Stakeholders are engaged throughout the lifecycle, and changes in requirements are wholeheartedly welcomed by the team.  A changed requirement late in the lifecycle is a competitive advantage, as long as you can act on it.

For a more detailed discussion, we suggest you consider the article Disciplined Agile Data Warehousing.

In the past few years more and more teams have started to adopt agile approaches to data warehousing (DW) and business intelligence (BI).  We have been at the forefront of this movement, having developed foundational techniques around agile data modelling, database refactoring, agile database testing, and many more  that enable teams to be more agile in their approach to DW/BI development.  This blog posting overviews the differences between how a traditional team and a disciplined agile team team approaches DW/BI development.  To understand these differences, the following diagram compares the typical level of expended effort creating artifacts on traditional and disciplined agile teams.

There are several interesting differences to between the approaches. Disciplined agile DW teams will:

1. Create a high-level conceptual model early. A high-level conceptual model, or more accurately diagam at this point, identifies the critical business entity types and the relationships between them. This provides vital insight into the domain while helping to capture key domain terminology, thus helping to drive consistency of wording in other artifacts (such as user stories and epics). Traditional teams will often make the mistake of over documenting the conceptual model early in the lifecycle, injecting delay into the team (with the corresponding opportunity cost of doing so) as well as the risk of making important decisions when you and your stakeholders have the least knowledge of the actual end goal.
2. Evolve a light-weight logical data model (LDM) over time. If your agile DW team does this at all they will keep their LDM very light weight. Traditional teams will often invest heavily in their LDMs as they believe it is a mechanism to ensure quality and consistency through specifying it in. This often proves to be wishful thinking in practice. Disciplined agile teams instead invest their efforts in creating an executable specification in the form of regression tests (more on this below).
3. Evolve a detailed physical data model (PDM) over time. Disciplined agile teams realize that a PDM, when created via a data modeling tool with full round-trip engineering (it generates schemas as well as imports existing schemas), effectively becomes the source code for the database. As the requirements evolve the team will evolve the PDM to reflect these new needs, generating schema changes as needed. They can work this way because they are able to easily refactor and regression test their database. This is different from the traditional approach where they often perform detailed modeling up front. This is motivated by the mistaken belief that production database schemas are difficult to evolve, something that agilists know not to be true.
4. Develop a comprehensive regression test suite over time. These tests address several important issues. First, they validate the work of the team, showing that their work to date fulfills the requirements as they’ve been described to the team. Second, a regression test suite enables the team to safely evolve their work. Agile developers can make a small change, rerun their regression tests, and see whether they broke something (if so, then they either rollback their change or they fix what they broke). Third, when a test is written before the corresponding database schema or database code is developed, the test effectively becomes a detailed executable specification. Sophisticated agile DW teams will capture the kinds of information that were previously captured in LDMs in executable tests, and are thus much more likely to have consistent schemas than teams that still rely on static LDMs.
5. Capture critical meta-data over time. Because the rest of your organization may not be completely agile there is often a need to continue to capture key meta data about data sources. This meta data should be kept as light as possible. If there isn’t a definite need for it then don’t capture it. If someone says “but we might need it someday” then wait until someday and invest in capturing the information at that point. Furthermore, instead of capturing meta data in a static manner (i.e. as documentation) try to identify ways to capture it as tests, or to generate it automatically from other information sources. Any documentation that you write today needs to be maintained over time, slowing you down.

This blog posting is an excerpt from the detailed article Disciplined Agile Data Warehousing.