Disciplined Agile Applied

For years I’ve been involved with estimating a wide variety of things, have read about and then experimented with different estimation strategies, and taught and coached others in estimation.  I’ve worked with, and learned from, hundreds of people who also have a wide range of estimation experience.  Throughout all this I’ve identified several things that I believe to be true about the act of estimation.

In my experience, the quality of an estimate increases when:

1. The work is small. For example, it’s easier to estimate the amount of liquid in your water glass than it is to estimate the amount of water in the ocean.
2. The work is near term. We have greater motivation to think through an estimate when we’re just about to do something than if that work will be done at some point in the future.
3. The work is understood. Imagine someone coming up to you and saying “I’d like you to update the web site, how long do you think that will take?” The request is far too vague to respond with a reasonable estimate. Until you identify what that update is, there’s a big difference between fixing a grammar mistake on a page and adding an ecommerce marketplace to sell products on your site, you really can’t estimate the work.  Furthermore, you also need to understand the architecture of your site and the work processes involved with updating it.  Fixing and deploying a grammar mistake may be five minutes of effort or five hours of effort depending on your environment.
4. The work is stable. When the requirements for the work change so must the estimate. What seem to be simple requests, such as asking for multi-currency support in your ecommerce marketplace when you originally said you only needed to support a single currency, can be an expensive and time-consuming change.
5. You know who is going to do the work. People are unique, they aren’t fungible resources that you can easily swap.  For example, when building a fence, a team of three people who build fences for a living will very likely get it done faster, better, and for less cost than a team of three people who are building a fence for the first time.  A master craftsperson with years of experience may be 10 times or even more productive than a novice.  In software development, studies have shown that the most productive programmers are 25x more effective than the least productive programmers.  And we all know that some people just aren’t going to work well with others, so team configurations are also important considerations.
6. The process and tools are understood. When estimating how long it will take to dig a hole for the foundation of a house, it would be good to know if it’s being dug by three people with shovels or one person with a backhoe.  
7. The estimator is also the doer.  Someone who is going to do the work is much more motivated to get an estimate right than someone who isn’t going to do the work. 
8. The estimate is being committed to.  When you know that you’re going to be held to the estimate that you produce you are much more motivated to get it right.
9. The estimator has done this sort of work before.  People who have done the work before have a better understanding of the issues involved than people who haven’t.  
10. There are several estimates being combined.  In high-risk situations I strive to estimate something using several different strategies, ideally involving different people, and then combine the results to get a better quality estimate.
11. It is treated as a probability distribution. By treating estimates as probability distributions (which they are), and not as point-specific scalars, we can apply our estimates in a realistic manner.

The further away you are from these truisms the less dependable an estimate will be.  As the old saying goes, “garbage in, garbage out.”.  In the next blog in this series we will explore the trade-offs associated with estimation.

A common practice is to present estimates in terms of ranges, such as $500K to $900K, rather than point-specific scalars such $650K.  The size of the range that you provide reflects the quality of your understanding of what you’re estimating. PMI’s Practice Standard for Project Estimating (2^nd Edition) suggests an initial guesstimate of -25% to +75%, in general, on projects. We want to present our estimates in ranges so as to set realistic expectations with our stakeholders. 

Your mileage may vary, sometimes dramatically. Barry Boehm, one of the most respected researchers in the software engineering world, has found that a range of -75% to +300% occurs in practice on software projects (see Software Engineering: Barry W. Boehm's Lifetime Contributions to Software Development, Management, and Research).  The initial range is so large because our stakeholders often don’t know what they want, or can’t agree on what they want, and thus it makes it very difficult to estimate.  As Boehm shows this problem is particularly acute in the software world due to the flexibility of software, expectations around software, and the unfortunate tendency of software professionals to agree to requested changes (and then delivering on those promises in sometimes questionable manners).  Marc Bastien presents an insightful exploration of these challenges in his recent book Understanding the Corporate IT Strategy Game.

Figure 1 is based on an actual software development project taking a “predictive” serial approach where the original estimate of 10 months was based on an up-front function point (FP) count.  This is depicted by the first line in Figure 1, as a point-specific scalar estimate.  But as we know estimates are probability distributions and should therefore be presented as a range.  The second line in Figure 1 shows the initial estimate as a probability distribution.  Although it appears to have a large range, which can be disconcerting to anyone hoping for a scalar estimate, this team was able to tighten the range to be 9 to 13 months due to the significant investment they made in function point counting. Over time, as our understanding improves and the uncertainty shrinks the range of the estimate also shrinks.  

Figure 1. Ranged estimates will narrow over time on a healthy project.

An interesting aspect of Figure 1 is that after 3 months you can see that the range of the estimate has tightened a bit and has shifted to the right (the project is likely to take longer than originally believed).  Notice how I said likely to take longer and not going to take longer – that’s because estimates are probability distributions and not certitudes, so I choose to be careful with my language and not imply that the estimate is precise.  Similarly, at the six month point we’re seeing a much narrower range, that’s because we have a much better understanding of both the problem we’re hoping to solve and the solution that we’re implementing to do so.  Figure 1 indicates that the likely project delivery date is clearly slipping over time, with a peak of about 10 months, 13.5 months, and 14.5 months respectively.  Needless to say the project manager had a few testy conversations with the project stakeholders throughout this period.  In the end this team delivered at the 16 month point and then had a second release 6 months later to address quality problems and dropped scope as the result of the “rush job” required to release after 16 months.  So the 10 month project took 22 months in practice. 

Tracking Ranged Estimates Over Time

An estimation trend chart, see Figure 2, is a useful way to depict ranged estimates over time.  This diagram corresponds to what we saw in Figure 1, albeit with monthly rather than quarterly estimates.  The vertical bars represent the ranged estimate each month and the dashed lines are the smoothed trend lines for high and low ends of the ranges. These two lines indicate what is called the “cone of uncertainty,” an indication of relative risk surrounding the precision of your estimates over time.  On a healthy project the two lines will converge at the end of the project, on an unhealthy project that is trending towards failure they will not converge (more on this in a minute).

Figure 2. An estimation trend chart.

At the start of a project you will very likely begin with a guesstimate, which is an estimate with a very wide range.  Over time the range narrows and your guesstimates evolve into estimates.  The point at which guesstimates are considered estimates is up to your organizational culture.  In the case of the project depicted in Figure 2, I suspect that was around the fourth or more likely fifth month of the timeline.

I would be remiss if I didn’t point out that if your estimation range isn’t shrinking over time then that is a sign that your project is in trouble. There are several potential causes for this to happen, including a lack of agreement amongst stakeholders as to the scope of the effort, an inappropriate solution architecture/design strategy, or challenges around how your team is formed (typically you don’t have the right people or sufficient people to get the job done).  In Disciplined Agile (DA) we specifically address these common risks via the stakeholder agreement milestone, the proven architecture milestone, and explicit advice about forming teams respectively. In short, on healthy projects your ranged estimates will tighten over time and on unhealthy projects, if you’re being honest in your estimations, they very likely will not.

The fact that estimates should be presented in ranges is a critical concept for both your executives and your stakeholders to accept.  If they don’t understand the implications of this, and that these ranges really represent probability distributions, then they will not be able to govern effectively.  In the next blog in this series I’m going to share a collection of truisms about estimation that I believe you will find interesting.

In most situations it is a good practice to provide an estimate as a range instead of a fixed-point/scalar.  What I mean by this is that instead saying that a project will take ten months to complete is better to say that it will take between nine and thirteen months given our current understanding of the situation.  In theory the strategy of quoting estimates as ranges is great.  In reality we often run into several unfortunate challenges, which we explore in this blog.

The potential challenges that you face with ranged estimates in practice include: 

1. Some people still want fixed-point estimates. Some people want the “certainty” of a fixed-point estimate, rather than the vagueness of a range.  The idea of a ranged estimate, let alone the concept that an estimate is really a probability distribution, simply doesn’t fit into the way that they want the world to work.
2. Estimates are positively skewed probability distributions. We worked through this conceptIn my previous blog, Estimates Are Probability Distributions.  In Figure 1 the green line shows that this is a positively skewed distribution where there’s a much greater chance that a project will be late or over budget rather than being early or under budget.  The implication is that over time, as we update our estimates based on improved information, the estimate is likely to worsen.
3. People often want greater precision than is appropriate.  Even when people are willing to accept ranged estimates they will often ask for a narrower range than is warranted.  For example, PMI’s Practice Standard for Project Estimating (2^nd Edition) suggests that a project in the start-up phase may have a rough order-of-magnitude (ROM) guesstimate of -25% to +75%.  That’s a huge range that’s hard to sell to people.  In Figure 1 the red line depicts how people ask for a narrower range, such (+/- 25% or even +/- 10%) rather than the harsh reality depicted by the green line.
4. Stakeholders often choose to focus on the low end of the range. An unfortunate axiom of project management is that whenever you give a ranged estimate your stakeholders are likely to focus on the bottom of the range.  For example, tell them something will cost between $50 and $100 and they home in on $50.  Worse yet, they’ll start trying to get you to commit to an even lower figure because they think they can motivate you to do better. 
5. The team often chooses to focus on the high end of the range.  When you guesstimate that a project will take between four and ten months to complete they’ll often choose to believe they have ten months.  Experienced team members, having seen many projects go late or over budget, will believe they have even longer.
6. The historical data either isn’t there, isn’t sufficient, isn’t applicable, or isn’t applied properly. Whew! Thanks to Robin Goldsmith of Go Pro Management who wrote an insightful comment on Estimates Are Probability Distributions about this problem.  The challenge is that few organizations have sufficient historical data from which to base estimates and if they do have such data it is often from dissimilar projects.  Even when organizations do have historical information they often misapply it.

Figure 1. The practical realities of ranged estimates.

The good news is that by being aware of these challenges we can choose to overcome them and apply ranged estimates effectively.  One strategy of doing so is to start with a very wide range, effectively a guesstimate, at the beginning of a project and then as our understanding improves we will update to our estimate and very often narrow the range.   The next blog in this series will explore this approach.

Estimation is an important skill required by many professionals. My experience is that there are three distinct levels of how people think of estimates.  My goal is to focus on the third level, estimates as probability distributions, in this blog.

Let’s begin with a quick overview of the three estimation mindset levels:

1. Estimates as point-specific numbers.  For example, a point-specific estimate would be that this is a million-dollar project that will take six months to complete.  In the case both the cost and time estimates are fixed, scalar numbers. This mindset often stems from a desire to have an “exact number” on which to base decisions.  
2. Estimates as ranges. An example of a ranged estimate is that this project will cost between $800,000 and $1.4 million and take between 4 and 10 months to complete.  In fact, PMI’s Practice Standard for Project Estimating (2^nd Edition) suggests that a project in the start-up phase, what Disciplined Agile (DA) calls Inception, may have a rough order-of-magnitude (ROM) guesstimate of -25% to +75%.  On software projects, Barry Boehm has found that -75% to +300% occurs in practice (see Software Engineering: Barry W. Boehm's Lifetime Contributions to Software Development, Management, and Research).  Yikes.  This estimation mindset often stems from discovering that “exact” estimates often prove to be unrealistic in practice and as a result they need to be more flexible in practice.
3. Estimates as probability distributions.  This is an evolution of ranged estimates in that it reflects the observation that there is a chance that a project will come in below budget (or early) and that there is a chance that a project will come in over budget (or late). This is a strategy that I first learned from Murray Cantor, author of Software Leadership, when I worked with him at IBM Rational.  This estimation mindset tends to be held by people with a strong background in mathematics, in particular statistics.

Figure 1 depicts an estimate as a probability distribution.  It shows a point-specific/fixed estimate, such as $1 million, which is aligned with the peak of the curve.  The area under the graph to the left of the fixed estimate point is the chance that the project will come in under budget and the area to the right is the chance that the estimate will come in over budget.  This probability curve would be determined from past history of previous projects within your organization.  The curve in Figure 1 is “positively skewed” in that there is more area to the right of the curve’s peak than to the left.  The implication is that there is a greater chance of the project being over budget than there is of it being under budget.

Figure 1. Estimates as probability distributions.

Let’s work through how you actually apply this knowledge in practice.  Figure 2 shows an estimate distribution for a fictional project where we are quoting estimates in terms of probabilities.  For example, the chance that this project will come in at $900K or less is the area under the graph to the left of that line, in this case it looks like it’s about 8%.  The chance that it will come in at $1M or less looks to be about 22%, $1.1M or less about 45%, and $1.2M or less about 65%.  The implication is that depending on the level of certainty your stakeholders require you can use this graph to provide an estimate that meets their need.  If your stakeholders wanted 95% surety, for example, then you’d find the point on the estimate distribution where 95% of the area is to the left of the line, in this case it looks like it would be around $1.5 or $1.6 million.  This would be a lot easier, of course, if you were using software to come up with these sorts of numbers rather simply "eye-balling" a graph.  Once again, all of these numbers are based on the past history of previous projects within your organization.

Figure 2. Quoting estimates as probabilities.

There are several challenges with the concept that estimates are probability distributions:

1. Many people don’t understand the concept of probability.  For example, I just opened the weather app on my phone and it says that there is a 70% of snow this hour.  So, if it doesn’t snow in the next hour is the app right or wrong?  It’s right, because there was a 30% chance that it wouldn’t snow.  But what if the app said there was a 90% chance that it will snow, but then it doesn’t?  The app was still right.  The point is that many people will hear a probability such as 70% or 90% and assume that it’s a certainty (100%).  Conversely, they’ll hear that there is a low probability of an event and then assume that it won’t happen and won’t be prepared if it does. 
2. People who desire “predictability” don’t like this idea.  There are still people who prefer the “certainty” of a point-specific estimate, or at least an estimate with a very tight range.  While these expectations might be reasonable on straightforward projects they certainly aren’t on projects where stakeholder needs or implementation strategies evolve constantly (as is typical on software projects, for example).
3. Estimates don’t follow a normal bell curve distribution.  As I indicated earlier, estimates tend to follow a positively skewed distribution.  This is the result of overly-optimistic estimation or pressure to produce an estimate that is palatable to decision makers.  This positive skewing can be frustrating for many people because the implication is that you’re more likely to be over budget or late than you are to be under budget or early respectfully.  

In the next blog in this series we’ll work through the strategy of ranged estimates in greater detail.  

In The Art of Guesstimation: Why Should We Estimate? we explored the reasons for why it can make sense to estimate the cost or time that an endeavor will require.  But this isn’t the whole story.  An important principle of business analysis, and in project management for that matter, is to not only define what is in scope but also to indicate what is out of scope. The implication is that it also makes sense to explore why we shouldn’t estimate an endeavor, which is the topic of this blog. 

There are several reasons why we should consider reducing the amount of effort we put into estimation if not eliminating it entirely:

1. Estimation takes time and money. There is a cost associated with creating an estimate. As with any other activity, we want to perform the least amount of work possible that gets the job done and no more because anything more than that would be a waste.  The article The Value of Planning provides a detailed explanation of the importance of creating artifacts, in this case estimates, that are just barely good enough (JBGE).  
2. Estimation is an overhead.  From a lean point of view we want to reduce, and better yet eliminate, any waste in our process.  As the #NoEstimates movement is very clear about, estimation is an overhead that does not directly add value to a customer (except in the form of providing inputs into their decision making process). At a minimum we want to find ways to reduce the amount of effort that we put into estimation and ideally to eliminate it completely where possible. 
3. People commit to important decisions too early. The more effort that we put into producing an estimate, typically because we desire greater precision in the estimate, the more motivated we are to commit to whatever decisions that we make when developing the estimate.  For example, to develop a reasonably precise estimate for a software project we may need to invest several weeks, or even months, to identify the requirements and architectural solution for an endeavor.  The more work that is put into defining the requirements, or the architecture, the greater our reticence to evolve those decisions later in the project.  Even in the face of a different market environment, or an improved understanding of what our customers actually want, we will be wary of rethinking the commitment that they made earlier to the original requirements upon which the original cost and time estimates were produced.  In effect we’ve made very important decisions early in the project when our understanding of the problem, or the potential solution, was vague at best. In the end we may be on time and on budget, but we’ll have produced the wrong solution to the problem and thus have still failed in the eyes of our stakeholders. 
4. Estimates answer the wrong questions. A cost estimate tries to answer the question “How much do we believe X will cost?” and a schedule estimate tries to answer the question “How long do we believe X will take?”  I appreciate that this will sound like heresy to many people, but should we not be focused on more interesting questions than that?  For example, rather than focusing on cost should we not strive instead to spend our money wisely?  Is getting a low-quality product on budget really a win?  Particularly when we’ll need a second effort to fix the quality problems?  Similarly, rather than focusing on a delivery date should we not strive to deliver what our customers need when they actually need it?  Is getting something built to specification delivered on time really a win when the requirements evolved while we were building the product?  Would it not be better to let the date slip and produce something people actually want, or deliver a partial solution early that meets some of their needs right now?
5. Better ways of working (WoW) require less estimation.  You can dramatically reduce the effort required to estimate something, and even eliminate it completely, if you adopt effective WoW.  For example, consider a long-standing team (often called a product team) who are following Disciplined Agile’s (DA) Continuous Delivery: Agile lifecycle.  This is a team of fifteen people and they release into production every Wednesday morning at 11 am.  Estimating the delivery date for the team is easy, it’s “Wednesday at 11 am.”  Estimating the cost of that release is also easy, it’s fifteen times the charge out rate for one person for one week.  It’s at this point in the conversation that people say that it’s not that easy, because you still need to indicate when you’re going to deliver a specific thing.  Yes, you’re still going to need to do a bit of work prioritization and in the case of regulatory work or contractual work you’ll have to size your work somehow, prioritize it, and manage to the date.  So keep your sizing effort as simple as possible, or organize your work into similarly sized chunks, which also requires a bit of effort to accomplish, to enable this level of work management.  In short, improve your WoW and squeeze out the management bureaucracy.

The point that I’m making is that although we’ve seen that there are reasons why we what to estimate time and cost there are also reasons why we don’t.  A fundamental observation of the Disciplined Agile (DA) toolkit are that there are no “best practices,” that every strategy has strengths and weaknesses, that every strategy makes sense in some situations and not others.  This observation applies to estimation, just like it applies to everything else.

In the next blog in the series we will work through why estimates are probability distributions.