Game Theory in Management

When the US Department of Energy first came into existence, their guidance on performing project management was pretty much the same as the rigorous version used by the Department of Defense. However, it didn't take long for those requirements to get watered down via DOE Order 4700.1 and 4700.5, which implemented a so-called "graded approach." This graded approach spelled out the circumstances under which certain projects could opt out of certain aspects of the project management requirements, based primarily on the complexity and risk profile of the specific project. As soon as the new requirements came out -- wouldn't you just know it? -- a whole bunch of projects that had previously been considered appropriate for complete PM implementation were suddenly asserting that they were, actually, very simple, with low risk profiles! A veritable avalanche of arguments against projects having to comply with the complete set of PM requirements ensued, the vast majority of which were completely bogus. I was even involved with a project where the project manager sought to have his project renamed as a program, since programs, per se, were not required to obey the previous rules, only projects.

Flash forward to the Summer of 2009. Based on the release of my first book Things Your PMO Is Doing Wrong (PMI Publishing, 2008), I had been invited by PMI®’s Information Technology Special Interest Group to do a webinar on the difficulties of performing traditional project management techniques in an IT environment. The webinar, entitled “Stop Those Divorce Proceedings! Performing Earned Value Analysis in an Agile/Scrum Environment” was well-received, but I came across some disquieting factoids while I was doing my research. Of course, the frequency with which software projects come in over-budget and late is such that poor performance against their baselines had already become axiomatic. But, since 1986, when Hirotaka Takeuchi and Ikujiro Nonaka  published the seminal work that would form the basis for Agile/Scrum project management, the efforts to streamline project management to rid it of its more restrictive aspects in order to make it useful in an IT environment has led to a few “graded approach” moments which, in turn, threaten to return software projects to their days of persistent overruns and delays.

Take, for example, the aforementioned use (or lack thereof) of Earned Value Management Systems in IT projects. There is a myth, perpetrated by those who don’t like or understand project management in general and Earned Value in particular, that EV “requires” the existence of highly-accurate time-phased budgets, known in the EV world as “Planned Budget,” or, for us old-timers, the Budgeted Cost of Work Scheduled. Without these well-estimated and time-phased budgets, say the ignorant resistors, all subsequent Earned Value analysis is rendered useless. This lie is tailor-made for those who seek to avoid having to do any PM at all by asserting the Agile/Scrum approach. When, they ask, we are in the middle of a scrum, and the scope baseline has been changed and approved, does anyone have time to perform a bottoms-up estimate of the new scope? Isn’t that why Agile/Scrum was invented in the first place – to avoid software development teams having to stop work to formally reset the baselines in order to comply with some stodgy, old PMBOK Guide®-recommended requirement? Surely it’s best to eschew (actually, these Visigoths would never use a word like “eschew,” but hang with me) all Earned Value Methodologies for this work, and subsequent projects like it!

The truth here is that Earned Value does not need a highly-accurate time-phased budget to return extremely valuable management information. In fact, in those instances where the work has been woefully mis-estimated, Earned Value is the best analysis to uncover the error. Say you had a task that has been estimated to cost $100,000 (USD), but the more appropriate cost estimate would have been twice that. The task gets underway. At the end of the first reporting cycle, your project controls analyst asks your overall percent complete, and you inform her that it’s about 25%. However, your project controls analyst collects your actual costs, and they come in at $50,000. Using the traditional formula for calculating an Estimate at Completion, :

EAC = (BAC / CPI)

where BAC is the budget at completion ($100,000) and CPI is the Cost Performance Index (Earned Value divided by Actual Costs; in this example, 25,000 divided by 50,000, or 0.5), she instantly knows that this task will most likely cost $200,000 – which is what a perfectly prescient estimator would have known beforehand.

Okay, but what does this have to do with Agile/Scrum, and costing the change that was introduced in the middle of the last Scrum? The previously shown formula can be algebraically reduced to:

EAC = ACWPcum / % Complete

where ACWPcum is the cumulative amount spent on the task. In the example, $50,000 divided by 25% also produces the $200,000 amount. Simply ask the Task Manager to estimate his percent complete based on the expanded scope, and divide that figure into the altered task’s cumulative actual costs, and you have the new estimate, ready to plug into the cost baseline.

This is but one example, but there are many others where traditional PM techniques may be compromised safely, and others that cannot. How to differentiate? For that, the insightful reader will want to order my recently-released, must-have book, Game Theory in Management, and I look forward to y’alls’ comments.

As a continuation of the Olympic-sized meme and my blog from last week (“Pathologies in our Decision-Making”), I want to explore the notion that large populations are easier to predict than smaller ones. I first became aware of this idea as a boy, when I was reading Isaac Asimov’s Foundation trilogy. For those of you who are unaware of this classic of science fiction, in a future Milky Way Galaxy mankind has spread throughout, and an inter-stellar government of sorts is set up on the planet Trantor. One Hari Seldon invents a way of calculating how future events will unfold, and names it the Risk Management Special Interest Group from PMI®. No, just kidding. He names it “Psychohistory,” and, as one might imagine, many people are very interested in what Hari and his associates “know” is going to happen in the future.

Now, the way Psychohistory’s mathematical processes are (barely) described, one could not possibly predict the behavior of a single gas molecule. But, get a bunch of them together in a container, and suddenly their behavior becomes quite predictable. Since, at this point in the future, the number of humans in the galaxy is in the trillions, the macro-future can be calculated, even if the micro-future can’t be known with certainty.

One of the fascinating (and intellectually honest) things about the Foundation Trilogy is how the Seldon Plan is undone. A mutant arises, one who can telepathically change people’s minds. He quickly rises to power, and very nearly destroys the entire basis for inter-stellar civilization. The reason Psychohistory could not be used to see his rise to power coming is because he was a mutant, and such mutations were outside the model that Psychohistory used.

Certainly one of the drivers behind this bigger-is-easier-to-predict bizarro-world narrative lies with the notions of the statisticians, who attempt to use their data to calculate the odds of future events actually occurring. Small sample sizes, you see, are given to large variances. The larger the sample size, generally speaking, the more it assumes the familiar shape of the Bell Curve, should it be graphically represented. So, if some statistician were to, say, use the tools of statistics and probability to predict the outcome of this November’s presidential election, it just won’t do to ask a few dozen people at the campus of the University of California, Berkley, whom they would vote for (actually, the University of California anywhere). Our statistician would simply have to throw in an occasional meat-eating church-goer, if he wanted an accurate prediction.

But even with a good sample, there are (at least) two major problems with predicting the future based on data that captures past behaviors: the limiting effects of mathematical models, and network theory.

Notice how Psychohistory was undone because of an event that occurred that was outside of its model. Any attempt to limit possible behaviors, strategies, or tactics from any given population is extremely vulnerable to occurrences of the highly improbable, as Nassim Taleb describes brilliantly in his book The Black Swan, the Impact of the HIGHLY IMPROBABLE (Random House, 2007). But what about the notion that the sheer size of the sample somehow mitigates the impact of highly improbable (i.e., ”outliers”) events? This butts up against network theory, and Metcalfe’s Law.

As I insightfully discuss in my recently-released, must-have book Game Theory In Management, Metcalfe’s Law shows that while, say, two telephones have one connection between them, six phones have 15. In other words, the larger the network, the more powerful it is. Ironically, the larger networks become, the more susceptible they are to cascading events, also known as the Butterfly Effect, stated colloquially as “if a butterfly flaps its wings in Brazil, does that cause a hurricane in Texas?” Put another way, seemingly small variations in few (or singular) nodes in a large network can have catastrophic effects on a large number of nodes that are nominally far away, or non-intuitively connected. In other words, the larger the sample size (or network), the more difficult it is to predict its behavior, even regarding cataclysmic events.

Essentially, the only way the risk management aficionados can assert any calculated insight into the unfolding of future events is if they (conveniently) ignore both Black Swan theory and Metcalf’s Law, and hope that no one in their epistemological snake-oil audience knows about them.

Or have read this blog.

The title of this piece is a phrase turned by Nassim Taleb, in his best-selling book The Black Swan, The Impact of the HIGHLY IMPROBABLE (Random House, 2007). He discusses several such pathologies, but his strongest (in my opinion) arguments and condemnations are saved for managers and analysts who rely on the fields of probability and statistics. As I relay in my recently-released, must-have book, Game Theory in Management, Taleb goes so far as to say (on page 355 of the paperback version) “This proves that everything relying on ‘standard derivative,’ ‘variance,’ ‘least square derivation,’ etc. is bogus.”

Proceeding from his very well-argued points, I would like to draw the conclusion that virtually all of what passes for modern Risk Management theory is invalid, and ought to be abandoned.

Gantthead wanted its bloggers to take on Olympic-sized issues in July, and Risk Management is pretty darn huge. A Google search of “Risk Management Consultants” returned over 90M hits on July 14, 2012. Risk Management is one of nine chapters in PMI®’s hallowed Guide to the Project Management Body of Knowledge®. There’s even an ISO Standard for it.

To be clear, risk management does have a place in project management, but it’s much smaller than advertised.  Analysis methods Decision Tree and Monte Carlo simulation can provide a reasonable estimating parameter for calculating how much in funds reserve a given project should have, or identify appropriate targets for insuring risks. Past that, RM’s claims are pretty much overblown, and the axiomatic “80% confidence interval” is right out.

Consider the Drake Equation, often invoked to defend (and attract funds for) that fool’s errand, the Search for Extra-Terrestrial Life, or SETI. The formula looks like this:

N = N * fp ne fl fi fc fL

Where N is the number of stars in the Milky Way galaxy; fp is the fraction with planets; ne is the number of planets per star capable of supporting life; fl is the fraction of planets where life evolves; fi is the fraction where intelligent life evolves; fc is the fraction that communicates; and fL is the fraction of the plant’s life during which the communicating civilizations live. As Michael Crichton pointed out in his lecture “Aliens Cause Global Warming” (Caltech Michelin Lecture, January 17, 2003) nobody has any idea what any of these parameters might be. Even the first one – the number of stars in the Milky Way – could be anywhere in between 10 billion and 40 billion. For truly experienced risk managers, that’s a range of thirty billion, and that’s the parameter we have the best shot at knowing! The Drake Equation is nothing more than an invitation to speculate, placed in pseudo-scientific terms. As Crichton himself put it, “An equation that could mean anything means nothing.”

Now consider the formula for calculating a contingency fund using a single-tier Decision-Tree analysis:

Cn = BAC – [(E1$ * E1%) + (E2$ * E2%) + (En$ * En%)]

Where Cn is the contingency budget, BAC is the budget at completion, E1$ is the cost of possible event one occurring, E1% is the odds of event one occurring, all the way through event n, which is the last possible event included in the analysis. Where do the existence of (and cost/schedule estimates for) these “events” come from? Usually a risk analyst working with a subject matter expert, or the owner of the particular cost account or work package being analyzed. Do I have to say it? The Decision-Tree analysis, like the Drake Equation, is simply an invitation to speculate, with such speculations almost certainly being chocked-full of the analysts’ cognitive biases.

But you’ll never hear risk managers admitting as such. They love asserting that they can, to a certain degree, quantify the future events in a given project. And, of course, knowledge of the future is pure management gold, making any plausible claim of being able to capture such information extremely attractive. But, just as medieval alchemist tried to combine common materials with lead to create literal gold, risk managers seek to combine common information streams into a reliable narrative of how the future of a project should be expected to unfold. Of course, they can’t possibly, and no amount of statistical jargon can change that fact.

One last little stinger example: after reviewing the techniques in Project and Program RISK MANAGEMENT, A Guide to Managing Project Risks and Opportunities (PMI Publishing, 1976), I estimate that there is a 72% chance that the majority of the techniques in Project and Program RISK MANAGEMENT, A Guide to Managing Project Risks and Opportunities (PMI Publishing, 1976), will be considered as laughably obsolete as medieval alchemy within 45.2 years, with an 80% confidence interval. How did I come up with my variables (such as, how does one evaluate the comparable laughability of misguided medieval pseudo-scientific pursuits)?

Trust me – I’m an expert.

Yeah, I know this blog’s title sounds counter-intuitive. If “success” is a problem, than what in the world isn’t?

But stay with me a few sentences, and I promise to make things clear. I want to examine the term of a person “being the victim of their own success.” It usually applies to one who has been successfully implementing a certain tactic for so long that they no longer recognize new scenarios where the tactic is inappropriate, and then fail trying to implement this favorite tactic where it doesn’t belong. Put another way, when we enjoy success on a large scale, it’s very easy to attribute such an outcome with tactics, behaviors, people, and environmental factors that really had little (or even nothing) to do with the achievement being enjoyed. 

As Eric Berne wrote about in his classic book Games People Play (Grove Press, 1964), and I build on Dr. Berne’s ideas in my recently-released must-have book, Game Theory in Management, people tend to construct narratives, or scripts in their minds that serve a variety of purposes, including explaining who we are to ourselves and why our pasts have unfolded the way they did. Now consider the project team member or manager who has been associated with a project that, once completed, was considered a significant achievement and on a large scale. Unless such a person has an advanced understanding of the rules of evidence, logic, and causality (and in many cases, even if they do), odds are that significant parts of the narrative they embrace will contain cognitive biases, such as confirmation bias, which “informs” them of the characteristics of that achievement. Sometimes the association will be valid, and loosely-held. Other times, the association is invalid, and tightly-held.

For example, I was once providing the cost and schedule performance information for a very large and high-profile project. The Control Account Manager (or CAM) for the project management task had recently come off of another high-profile project that had come in on-time and under-budget. At the very first meeting on how we would set up the Earned Value and Critical Path systems, this CAM announced that my team’s first priority would be to generate … a “swim lane chart.” As I exchanged puzzled looks with my team, the CAM went to a white board and drew a set of parallel horizontal lines, and then place boxes representing tasks in between the lines. The “lanes” represented organizations within the project team.

“So, what you’re asking for is essentially a PERT chart, segregated by performing organization?” I asked. After he affirmed that that was what he was after, I offered “I think we need to develop a Work Breakdown Structure first, and then the cost and schedule baselines. Then we’ll be able to create your chart.”

The CAM was adamant. The “swim lane chart” had to be our first priority, period, and he would hear no talk of WBSs or baselines. Apparently, he had had access to these org-chart-arranged PERT charts in his earlier project, had become convinced that this particular format represented his information system silver bullet, and no amount of discussing the particulars of his request could dissuade him. Instead, the team had to essentially create the Work Breakdown Structure and cost/schedule baselines in secret, while maintaining publicly that we were exclusively pursuing the data elements to create the “swim lane chart.”

We were lucky, then. Other “successful” project managers aren’t so easy to work around. Their approaches to new problems can become chock-full of formulaic technical approaches, and any perceived deviation simply must be due to a lack of expertise or recalcitrance from the staff, neither of which is tolerable.

Conversely, managers coming off of poorly-performing projects, or even disastrous ones, tend to have greater respect for common but formidable problems that lead to overruns and delays. They also tend to retain that certain level of humility that leads them to at least listen to the other members of the project team, especially when different alternatives to approaching problem-solving are discussed. Of course, none of this is true when projects fail, and their decision-makers are not held responsible; but, in a free market environment, successful managers rarely repeat mistakes, whereas successful projects may have incurred many mistakes, but they were never identified, much less analyzed.

Andthat’s the problem with big success.

In honor of the Olympics beginning later this month, I would like to turn my readers’ attention towards very large project management problems; or, better stated, with management problems of very large projects. There’s just something about very large projects that, when they go South, they do so in dramatic fashion, like the Hindenburg or the Titanic. Interestingly enough, Titanic had a sister ship named Olympic, which distinguished herself by colliding with two different ships in her career, which is, I suppose, a superior fate to either of her sisters (the third ship in the class, Brittanic, was sunk during World War I after striking a mine). One of the ships she hit wasn’t even moving, being a lightship.

So, in examining the whole the-bigger-they-are-the-harder-they-fall motif, I would like to discuss the National Ignition Facility, or NIF. As I state in my just-released must-have book, Game Theory in Management; Modelling Business Decisions and their Consequences (http://www.gowerpublishing.com/isbn/9781409442417 ),

The NIF was a project designed to help researchers answer questions surrounding nuclear fusion. The concept was to surround a pellet of a hydrogen isotope with 192 high-powered lasers that could deliver sufficient energy to the target quickly enough to induce nuclear fusion, the same type of reaction that the Sun uses. The project was spearheaded by a scientist, who swore to Senate Appropriations Committee staffers that if Lawrence Livermore National Laboratory was selected to build the project, he would make sure it stayed on budget. Unfortunately, cascading risk events rendered that promise impossible to keep. Work began in 1997 with an estimated budget of $1.1 billion. The final price was nearly quadruple that amount. The Wikipedia article on NIF states:

The Pulsed Power Conditioning Modules (PCMs) suffered capacitor failures, which in turn caused explosions. This required a redesign of the module to contain the debris, but since the concrete structure of the buildings holding them had already been poured, this left the new modules so tightly packed that there was no way to do maintenance in-place. Yet another redesign followed, this time allowing the modules to be removed from the bays for servicing. Continuing problems of this sort further delayed the operational start of the project, and in September 1999 an updated DOE report stated that NIF would require up to $350 million more and completion would be pushed back to 2006.[1]

More problems like this manifested, but perhaps the most damaging risk event had to do with dust. High-energy lasers do not react well to dust: if the various mirrors and lenses used to aim and manipulate the beams have so much as a speck on them, the damage can be explosive, immediate, and expensive. Insufficient consideration was given to the difficulties of performing an extremely large construction effort, with new technology and large, heavy, and expensive components, all in a clean room environment. The early problems with design, construction, and dust cascaded to the point that NIF became the poster child for the perils of failing to perform the project management function adequately.

The truly ironic aspect to the NIF management disaster was the lateness with which management embraced Earned Value and Critical Path-based management information systems. The original estimate of $1.1 Billion (USD) would turn into $4.2 Billion by the time the construction part of the project was complete. Now, it may well be that had NIF management approached the U.S. Department of Energy with an initial estimate of $4.2 Billion, the DOE may have approved the project anyway. But that’s not what happened, leaving the typical management analyst to wonder what might have been. Even a simple Earned Value Management System would have provided early warning of the upcoming overruns, as well as the delays. But such basic project management information systems weren’t implemented in the early phases of the project. Management felt that they could get a handle on the cost performance of the project without an EVMS, with not-enough-life-boats or let’s-fill-the-dirigible-with-Hydrogen-sized consequences (well, to be honest, nobody died, but $3.1 Billion could have done a lot of life-improving, don’t you think?).

Of course, there’s always the possibility that, had a functioning EVMS been in place early-on at NIF, it would have had a similar impact to project controller Frederick Fleet calling NIF’s PM, and shouting “The Cost Performance Index-based Estimate at Completion is indicating a massive overrun, right ahead!”, and then watching helplessly as the failure unfolded. But, from an information technology point of view, it would have at least given some credence to the idea that management wasn’t flying completely blind. Being at the helm of a large project crash leaves those responsible with only a couple of narratives to further, neither of which is very attractive:

·         The “I didn’t see it coming, but then, nobody could have seen it coming” story line. The problem here is, if this kind of problem is out there, lying in wait for any similar effort, why would anyone undertake that kind of project in the first place?

·         The “I knew it was a possibility, but I thought our response to that event would have alleviated its negative consequences” script. This is easily overturned by noting either the number of lifeboats on board being far fewer than needed, or the fact that the lack of an EVMS in the earliest stages rendered project personnel hopelessly ignorant of performance-based cost overruns.

Of course, hindsight is 20-20, and there is a distinct element of unfairness to criticizing management decisions after-the-fact. I understand that, and do not wish to, as they say in American football, “pile on.” On the other hand, if you are (were) in charge of a large project that incurred massive overruns and delays, and you didn’t have a working Earned Value Management System from the get-go, maybe, just maybe you deserve a bit of being piled-on.

What do you think?