Categories: forecasting, research
“The majority of forecasters are fools or liars,” says Professor Bent Flyvbjerg from the BT Centre for Major Programme Management, at the Sa?d Business School, University of Oxford, in a new paper on inaccurate estimates for major projects.
The paper, published in the International Journal of Project Management, sees Professor Flyvbjerg criticising the way that forecasts for projects are put together. He says they are inaccurate and provide poor material from which to make decisions about cost and benefits.
“Estimates are commonly poor predictors of the actual value and viability of projects, and cannot be trusted as the basis for informed decision-making,” he says. “These forecasts frequently misinform decision makers on projects instead of informing them. Some of the inaccuracy comes from genuine forecasting mistakes arising from over-optimism, but some estimates are deliberately misleading, designed to secure financial or political support for a project.”
You probably know of examples of where a project manager has padded estimates for one reason or another, by Prof. Flyvberg is pretty scathing about forecasting methods and the people who use them.
“Many forecasts are garbage and can be shown to be worse than garbage,” he is quoted as saying in a press release from the university. “These reports give the client, investors and others the impression that they are being informed about future demand, or the costs involved in a major project, when they are being misinformed. Instead of reducing risk, reports like this increase risk by systematically misleading decision-makers and investors about the real risks involved.”
What’s the answer?
Prof. Flyvbjerg says that the answer is for everyone to be a bit better at not putting up with this (I paraphrase). For example, he recommends that clients should ask for their money back when they receive reports which later prove to be significantly inaccurate and misleading. He even goes as far as saying that they could demand compensation (some contracts must have a clause for this in anyway). His most radical idea is that in some cases criminal action would be justified. “Merely firing the forecaster may be letting them off too easily,” he says. “Some forecasts are so grossly misrepresented and have such dire consequences that we need to consider suing them for the losses incurred as a result. In a few cases where forecasters foreseeably produce deceptive forecasts, criminal penalties may be warranted.”
Personally, I can’t see many project managers ending up in court because of poor scheduling, but as this has come from the Centre for Major Programme Management, Prof. Flyvbjerg is really talking about complex, mega projects.
When we say ‘everyone’, Prof. Flyvberg includes the professional bodies in that too. He calls on them to use their codes of ethics to penalise and possibly exclude members who produce unethical forecasts. “This needs to be debated openly within the relevant professional organisations,” he says. “Malpractice in project management should be taken as seriously as malpractice in other professions like medicine and law.” How many project managers genuinely produce unethical forecasts and how many are just incompetent? I think it would be hard to decide if someone was acting in good faith and to the best of their abilities or whether they were deliberately altering estimates for political gain.
A better way of forecasting
As you would hope from someone who is so outspoken about this, Prof. Flyvberg has all the answers. His answer is to turn to his own work and in this IJPM paper he sets out the case for quality control and due diligence to be applied to the evaluation of front-end forecasts. Unfortunately, I think his answer only works for massive projects and not for the type of forecasting and estimating most project managers do on their projects.
“Recent research has developed the concepts, tools and guidance on incentives that could help curb both delusional and deceptive forecasts,” he says. “Whether forecasters are unwittingly or deliberately under-estimating the costs, completion times, and risks of projects, and over-estimating their benefits, we need to have a systematic basis for evaluating their findings in order to make informed investment decisions. Given the high cost of major infrastructure projects, the irreversibility of decisions, and the limited availability of resources, this is clearly critical for both public sector and private sector projects. Significantly more accurate forecasts can be produced by looking at the evidence available from previous similar projects which have been already completed – what I call, taking an ‘outside view’. This seems so simple, but in practice it is transformative and leads to much more accurate forecasting.”
In other words, take large data sets or statistically relevant data for projects in your sector, apply due diligence, estimate from the basis of past experience and critically evaluate the forecasts. You’ve spotted it – the big downside to this estimating approach is that you need large, validated data sets to draw benchmark data from previous, relevant, projects. If your PMO has been up and running for years and has gathered all this, and you never do any projects which innovate or deliver something new in a way you haven’t done before, then you could make use of this technique.
If you don’t have all that data to hand, then this method of forecasting will not scale from mega projects and programmes to the humble projects that you and I work on. While using historical data is great and we should all look to the past to better predict the future, we would be wrong to expect this model to work for all projects.