Disruptive Data, Or Time To Pound The Table?

 

I recently attended a conference on Predictive Analytics and Machine Learning but found it, for the most part, rather disappointing. Two of the presentations that I attended really stood out, for different reasons. The first – a truly superb one – both demonstrated an advanced knowledge in the field, and passed along usable insights. It was presented, if I’m remembering correctly, by a professor from Northeastern University. The other was just a mess. It was an attempt to attach value judgements to the results of an analysis model dealing with sentencing and parole decisions. I found the entire premise to be vacuous, for the following reasons:

• For MISs that inform public policy, no model makes decisions. They can attempt to inform decision-makers, and the decision-makers may adhere to a template, but the models, themselves, do not make these decisions.
• Data can be complete or incomplete, relevant or irrelevant, but absolutely cannot be legitimately labelled morally right or morally wrong.
• Similarly, the methodology used to process the data into usable information (alternately referred to as “the model” or “analytics,” as in “predictive analytics”) can be valid, or invalid. It cannot be considered morally right or morally wrong.
• It was clear that the presenters who had analyzed this model were doing so from an incomplete data set, and had no idea that that was their problem, much less how to pursue a remedy.

Recall Hatfield’s Incontrovertible Rule of Management # (I lost track), that all valid Management Information Systems –even those laying claim to performing “predictive analytics” – have the same high-level structure, to wit:

• Data is collected, based on some discipline or set of parameters. In PM, we tend to want to know the scope of the work, and the amount of resources and time needed to accomplish it.
• The Data is processed into usable information, usually based on some sort of methodology. For we PM-types, Earned Value and Critical Path methodologies are central to our Management Information Systems (MISs).
• The information is delivered to the decision-makers in a way that they can actually understand and make use of it. Entry-level Project Controls specialists often err when they believe that all PMs can instantly understand a Gantt Chart, or a Cost Performance Report in Format 1.

It quickly became clear to me and my associates that, while the paper presenters and organizations manning booths in the Exhibitor’s Hall were demonstrably adept at collecting data and then using that data to better identify the buying habits of specific demographics (the better to target marketing efforts), they fell quizzically silent when the topic moved away from anticipating purchasing trends. The favorite “predictive” tool of our friends, the risk managers (no initial caps), the Monte Carlo analysis technique, was never mentioned in my hearing. An old saw in the Legal Profession is “If the facts are against you, argue the law. If the law is against you, argue the facts. If the law and the facts are against you, pound the table.”[i] I think the management science derivative would be “when you don’t have the actual algorithm at the heart of you MIS down, talk about the data. When you don’t have a real shot at collecting the comprehensive data set to make your model work, talk about the algorithm. When you have neither, fake it.”

Speaking of faking it, a common but utterly invalid method for creating a Management Information System that supposedly gives advanced warning of problems is to set up some form of a common data repository, call it something like an “action item manager,” and then have it collect data without a clearly-defined discipline or structure. The resulting “system” is, essentially, a poll, a data stack surrounded by input and output nodes. The problem with a poll is that someone always has better (more recent, or more accurate) data, so that its holdings are unreliable. Add to that the fact that massive amounts of subjectivity get shoe-horned into this “system,” and you have an environment perfectly suited to the generation of an invalid MIS, passing along highly subjective data as reliable information. What could go wrong?

Which gets us back to the whole notion of data being “disruptive.” Data, by definition, can be timely or late, relevant or irrelevant, but it can’t be wrong, or inaccurate (the term “bad data” is analogous to “bad facts.”). If a particular fact is found to be disruptive, the question has to be asked, disruptive to what? Or to whom? Well, to a particular belief, and the person holding that belief. In this sense, the whole of the PMBOK Guide® could conceivably be seen as disruptive to all of those business schools still teaching the obsolete concept that the purpose of all management is to “maximize shareholder wealth,” similar to the fact that the Earth casting a round shadow on the moon during a lunar eclipse is “disruptive” data to flat-earthers. Is your model or analysis technique delivering unexpected or unreliable output? It could be disruptive data, sure. Verify the data, of course, but suspect your analysis technique.

Or you could just pound the table.

[i] Sandburg, Carl, retrieved from https://www.goodreads.com/quotes/918291-if-the-facts-are-against-you-argue-the-law-if on July 4, 2022, 17:38 MDT.