For years I’ve been involved with estimating a wide variety of things, have read about and then experimented with different estimation strategies, and taught and coached others in estimation. I’ve worked with, and learned from, hundreds of people who also have a wide range of estimation experience. Throughout all this I’ve identified several things that I believe to be true about the act of estimation.
In my experience, the quality of an estimate increases when:
The further away you are from these truisms the less dependable an estimate will be. As the old saying goes, “garbage in, garbage out.”. In the next blog in this series we will explore the trade-offs associated with estimation.
A common practice is to present estimates in terms of ranges, such as $500K to $900K, rather than point-specific scalars such $650K. The size of the range that you provide reflects the quality of your understanding of what you’re estimating. PMI’s Practice Standard for Project Estimating (2nd Edition) suggests an initial guesstimate of -25% to +75%, in general, on projects. We want to present our estimates in ranges so as to set realistic expectations with our stakeholders.
Your mileage may vary, sometimes dramatically. Barry Boehm, one of the most respected researchers in the software engineering world, has found that a range of -75% to +300% occurs in practice on software projects (see Software Engineering: Barry W. Boehm's Lifetime Contributions to Software Development, Management, and Research). The initial range is so large because our stakeholders often don’t know what they want, or can’t agree on what they want, and thus it makes it very difficult to estimate. As Boehm shows this problem is particularly acute in the software world due to the flexibility of software, expectations around software, and the unfortunate tendency of software professionals to agree to requested changes (and then delivering on those promises in sometimes questionable manners). Marc Bastien presents an insightful exploration of these challenges in his recent book Understanding the Corporate IT Strategy Game.
Figure 1 is based on an actual software development project taking a “predictive” serial approach where the original estimate of 10 months was based on an up-front function point (FP) count. This is depicted by the first line in Figure 1, as a point-specific scalar estimate. But as we know estimates are probability distributions and should therefore be presented as a range. The second line in Figure 1 shows the initial estimate as a probability distribution. Although it appears to have a large range, which can be disconcerting to anyone hoping for a scalar estimate, this team was able to tighten the range to be 9 to 13 months due to the significant investment they made in function point counting. Over time, as our understanding improves and the uncertainty shrinks the range of the estimate also shrinks.
Figure 1. Ranged estimates will narrow over time on a healthy project.
An interesting aspect of Figure 1 is that after 3 months you can see that the range of the estimate has tightened a bit and has shifted to the right (the project is likely to take longer than originally believed). Notice how I said likely to take longer and not going to take longer – that’s because estimates are probability distributions and not certitudes, so I choose to be careful with my language and not imply that the estimate is precise. Similarly, at the six month point we’re seeing a much narrower range, that’s because we have a much better understanding of both the problem we’re hoping to solve and the solution that we’re implementing to do so. Figure 1 indicates that the likely project delivery date is clearly slipping over time, with a peak of about 10 months, 13.5 months, and 14.5 months respectively. Needless to say the project manager had a few testy conversations with the project stakeholders throughout this period. In the end this team delivered at the 16 month point and then had a second release 6 months later to address quality problems and dropped scope as the result of the “rush job” required to release after 16 months. So the 10 month project took 22 months in practice.
Tracking Ranged Estimates Over Time
An estimation trend chart, see Figure 2, is a useful way to depict ranged estimates over time. This diagram corresponds to what we saw in Figure 1, albeit with monthly rather than quarterly estimates. The vertical bars represent the ranged estimate each month and the dashed lines are the smoothed trend lines for high and low ends of the ranges. These two lines indicate what is called the “cone of uncertainty,” an indication of relative risk surrounding the precision of your estimates over time. On a healthy project the two lines will converge at the end of the project, on an unhealthy project that is trending towards failure they will not converge (more on this in a minute).
Figure 2. An estimation trend chart.
At the start of a project you will very likely begin with a guesstimate, which is an estimate with a very wide range. Over time the range narrows and your guesstimates evolve into estimates. The point at which guesstimates are considered estimates is up to your organizational culture. In the case of the project depicted in Figure 2, I suspect that was around the fourth or more likely fifth month of the timeline.
I would be remiss if I didn’t point out that if your estimation range isn’t shrinking over time then that is a sign that your project is in trouble. There are several potential causes for this to happen, including a lack of agreement amongst stakeholders as to the scope of the effort, an inappropriate solution architecture/design strategy, or challenges around how your team is formed (typically you don’t have the right people or sufficient people to get the job done). In Disciplined Agile (DA) we specifically address these common risks via the stakeholder agreement milestone, the proven architecture milestone, and explicit advice about forming teams respectively. In short, on healthy projects your ranged estimates will tighten over time and on unhealthy projects, if you’re being honest in your estimations, they very likely will not.
The fact that estimates should be presented in ranges is a critical concept for both your executives and your stakeholders to accept. If they don’t understand the implications of this, and that these ranges really represent probability distributions, then they will not be able to govern effectively. In the next blog in this series I’m going to share a collection of truisms about estimation that I believe you will find interesting.
In most situations it is a good practice to provide an estimate as a range instead of a fixed-point/scalar. What I mean by this is that instead saying that a project will take ten months to complete is better to say that it will take between nine and thirteen months given our current understanding of the situation. In theory the strategy of quoting estimates as ranges is great. In reality we often run into several unfortunate challenges, which we explore in this blog.
The potential challenges that you face with ranged estimates in practice include:
Figure 1. The practical realities of ranged estimates.
The good news is that by being aware of these challenges we can choose to overcome them and apply ranged estimates effectively. One strategy of doing so is to start with a very wide range, effectively a guesstimate, at the beginning of a project and then as our understanding improves we will update to our estimate and very often narrow the range. The next blog in this series will explore this approach.
Estimation is an important skill required by many professionals. My experience is that there are three distinct levels of how people think of estimates. My goal is to focus on the third level, estimates as probability distributions, in this blog.
Let’s begin with a quick overview of the three estimation mindset levels:
Figure 1 depicts an estimate as a probability distribution. It shows a point-specific/fixed estimate, such as $1 million, which is aligned with the peak of the curve. The area under the graph to the left of the fixed estimate point is the chance that the project will come in under budget and the area to the right is the chance that the estimate will come in over budget. This probability curve would be determined from past history of previous projects within your organization. The curve in Figure 1 is “positively skewed” in that there is more area to the right of the curve’s peak than to the left. The implication is that there is a greater chance of the project being over budget than there is of it being under budget.
Figure 1. Estimates as probability distributions.
Let’s work through how you actually apply this knowledge in practice. Figure 2 shows an estimate distribution for a fictional project where we are quoting estimates in terms of probabilities. For example, the chance that this project will come in at $900K or less is the area under the graph to the left of that line, in this case it looks like it’s about 8%. The chance that it will come in at $1M or less looks to be about 22%, $1.1M or less about 45%, and $1.2M or less about 65%. The implication is that depending on the level of certainty your stakeholders require you can use this graph to provide an estimate that meets their need. If your stakeholders wanted 95% surety, for example, then you’d find the point on the estimate distribution where 95% of the area is to the left of the line, in this case it looks like it would be around $1.5 or $1.6 million. This would be a lot easier, of course, if you were using software to come up with these sorts of numbers rather simply "eye-balling" a graph. Once again, all of these numbers are based on the past history of previous projects within your organization.
Figure 2. Quoting estimates as probabilities.
There are several challenges with the concept that estimates are probability distributions:
In the next blog in this series we’ll work through the strategy of ranged estimates in greater detail.
In The Art of Guesstimation: Why Should We Estimate? we explored the reasons for why it can make sense to estimate the cost or time that an endeavor will require. But this isn’t the whole story. An important principle of business analysis, and in project management for that matter, is to not only define what is in scope but also to indicate what is out of scope. The implication is that it also makes sense to explore why we shouldn’t estimate an endeavor, which is the topic of this blog.
There are several reasons why we should consider reducing the amount of effort we put into estimation if not eliminating it entirely:
The point that I’m making is that although we’ve seen that there are reasons why we what to estimate time and cost there are also reasons why we don’t. A fundamental observation of the Disciplined Agile (DA) toolkit are that there are no “best practices,” that every strategy has strengths and weaknesses, that every strategy makes sense in some situations and not others. This observation applies to estimation, just like it applies to everything else.
In the next blog in the series we will work through why estimates are probability distributions.