Failing by Irrelevant Comparisons

 

As I have stated (on multiple occasions) in this blog, the worst 20% of managers who are in possession of 80% of the information they need to obviate the best decision in any given circumstance will outperform the top 80^th percentile of managers who only have 20% of the information they need to illuminate the circumstances of the decisions before them. It follows, then, that any discussion on IT project requirements must include some sort of analysis that answers the question, which information?

I once worked with a VP who had a Ph.D. in statistics, and simply loved combing through page after page of the actual costs the projects were incurring, by line item, and comparing this data to the basis of estimate that made up that project’s cost baseline. Any deviation, of course, demanded an investigation. I tried to reason with him.

“David, consider the scenario where a $100K project baseline had $25K in labor costs, and $75K in heavy machinery. At the end of the project, the actual costs were $75K in labor costs, and only $20K in heavy machinery. By your analysis technique, a major problem has occurred, when, in reality, the project came in under budget.” But he refused to budge.

He wasn’t very successful.

I am, however, not without sympathy. As I discuss in my recently-released, must-have book, advocates of specific management information streams will push their products way, way past their epistemological limits, promising that, should the manager adapt their narrative, the path to perfectly informed management decisions will be there for the taking. This is not how management science theories are tested and validated – it’s how products are sold. And, when salesmanship and marketeering are allowed to trump legitimate management science research, organizational chaos ensues. Here’s a short list of a few of the pseudo-management science techniques that have wormed their way into Project Management Offices everywhere, and yet are, without question, invalid.

The Bottoms-Up Estimate at Completion (EAC). This is the PM technique of re-estimating the remaining work on a project, adding this figure to the cumulative actuals, and, shazaam!, you have a valid EAC. Except that you don’t.  According to AACEI, a “detailed” estimate, generated by a professional estimator using off-the-shelf estimating software, can only reliably provide an estimate accurate to within 15%, best case. As I have previously noted, using the formula EAC = Total Budget / Cost Performance Index gets you to within 10% on a consistent basis, and is far, far easier an analysis to perform. And yet, the “bottoms-up” aficionados have this insipid little technique firmly entrenched in many U.S. Government procedures, orders, and policies. It’s really just make-work for the organization’s estimators once the cost baselines have been approved.

Comparing Budgets to Actuals. As I pointed out in the previous story, this technique is not only invalid, but it returns misleading information. However, since our friends the accountants like to pretend that the general ledger must be THE source and residence of any and all management information that has to do with costs, this silly technique remains in all unenlightened managers’ tool kits. It’s so intuitive, though, isn’t it? Of course, the valid (and infinitely superior) way of relaying a cost variance is to compare the cumulative Earned Value (% complete * total budget) to cumulative actual costs, but the invalid version will never really go away. This information stream is just make-work for the accountants, when they’re not out nagging others about other irrelevancies.

Calculating the Cost Baseline’s 80% Confidence Interval. Of course, the risk management types as well as the statisticians just love performing this analysis, but it’s as useless as a sundial in Seattle. All of the assumptions about alternatives that go into either a Monte Carlo or a Decision-Tree analysis are nothing more than guesses by the project team. And quantifying and processing data based on guesses yields an information stream that is simply over-processed and over-thought guesses. I would normally wrap up this paragraph by saying that this technique is just make work for the risk analysts, but virtually everything these guys do is make-work, so why just pick on the 80% confidence interval?

If your customer accuses your project of failing due to one of these metrics, or, worse still, if performance against these techniques is called out in your requirements, then the good news is that your customer isn’t very advanced in valid PM techniques. But if your project team is doing this to themselves, then the bad news is … your team isn’t very advanced in PM techniques, and your odds of succeeding just dropped.

Do not ask me the exact percentage of how the odds of success have dropped. That piece of irrelevant information can be pulled out of thin air (strikethrough) generated by your risk analysts.