Reference Class Forecasting Depends on How You Define “Similar”
| Reference Class Forecasting (RCF), popularized by Bent Flyvbjerg, is used to address optimism bias in project cost and schedule estimation. The logic is simple: compare your project to similar completed projects and adjust expectations based on their actual outcomes. It is one of the most influential forecasting innovations in project governance. But a foundational question that requires more attention is: how is similarity defined in the first place? In practice, reference classes are often formed using broad administrative categories such as “light rail,” “metro,” or “heavy rail.” Yet these similarity criteria are frequently underspecified, even though they determine the empirical distribution from which percentile uplifts are calculated. Before any statistical adjustment occurs, a methodological decision has already shaped the forecast. In my recent study using a dataset of U.S. mass transit projects, I examined the sensitivity of RCF to alternative reference class formation. Rather than relying on predefined categories, I applied unsupervised clustering techniques to construct alternative reference classes based on structural attributes such as track length, number of stations, underground proportion, and rolling stock. Cost and schedule outcomes were excluded to preserve the outside-view logic. The results showed that changing similarity groupings altered cost and schedule distributions. Percentile-based uplifts shifted, dispersion patterns changed, and the implied contingency requirements varied across alternative clusters. In other words, RCF outcomes proved structurally contingent on how similarity was operationalized, not merely on statistical adjustment. This does not undermine RCF. The behavioral foundations established by scholars such as Daniel Kahneman and Amos Tversky remain essential. The outside view is still one of the strongest correctives to optimism bias in major projects. However, the findings suggest that the credibility of RCF depends not only on selecting the appropriate percentile (P50, P80, etc.), but also on transparently justifying how the reference class was formed. For practitioners and governance bodies, this has important implications. Reference class formation should be treated as a methodological decision requiring documentation and testing. Specifically:
|
Posted on: May 04, 2026 08:00 AM
| Permalink |
Comments (2)
The AI Readiness Problem in Project Management
| A mismatch is emerging in many AI-enabled project environments, and it’s not a technology problem. It’s a readiness problem. Today’s AI tools are impressive and accelerating in capability. We can model complex trade-offs, optimize schedules across thousands of constraints, cluster historical projects into meaningful reference classes, and flag budget risks earlier than any human. From a technical perspective, many remarkable capabilities are already available. Are organizations ready to take advantage of the technology? AI is advancing faster than the normal project decision process. Teams are given powerful models, only to be surrounded by old governance structures, incentives, and habits that existed long before AI appeared. The predictable result is that sophisticated analytics collide with an outdated decision environment. A common example is how AI outputs are framed. Many tools present a single “best” answer, represented as the optimal schedule, the lowest-cost plan, or the recommended portfolio priorities. This approach may be technically defensible, but it is behaviorally risky. When results are framed as answers rather than inputs, discussion shuts down. Judgment is replaced by deference, and responsibility quietly shifts from the decision maker to the algorithm. Another gap in organizational readiness lies in expectations. Organizations often expect AI to remove uncertainty, bias, or political tension from decisions. In reality, AI tends to expose these factors. Models surface uncomfortable trade-offs, inconvenient comparisons, and outcomes that challenge prior commitments. If leaders aren’t prepared for that friction, the model gets ignored or worse, selectively used to justify decisions already made. There’s also a skills mismatch. Not technical skills, but decision skills. Many teams are trained to use analytical tools rather than to interrogate assumptions, compare scenarios, or explain why one option was chosen over another. AI doesn’t eliminate the need for those capabilities. It makes good decision-making skills even more critical. The irony is that none of this requires better algorithms. It requires better integration, clear decision ownership, and explicit governance. Organizations need a cultural shift that treats AI as a strong opinion rather than a verdict. The real challenge with AI in projects isn’t what the technology can do. It’s about whether organizations are ready to let it inform judgment rather than replace it. Closing the gap between capability and readiness is where organizations can unlock the greatest value from AI. |
Posted on: April 27, 2026 08:00 AM
| Permalink |
Comments (1)
Why Blocking AI in the Workplace Doesn’t Work
| Some organizations still attempt to block large language models (LLMs) like ChatGPT from their networks in an effort to control the use of artificial intelligence (AI) at work. While the intention may be to reduce risk, the reality is that this approach rarely works. AI is simply too accessible. If an employee cannot access AI tools on their work computer, they can easily use them at home or on a personal device such as a smartphone. Many generative AI tools are free or offer free versions. A professional facing a complex problem, whether writing a report, analyzing data, or brainstorming solutions, can open a browser at home and ask an AI system for help in seconds. The same dynamic exists in machine learning. Programming languages such as Python are free (Open Source) and software development environments like Anaconda allow anyone to install powerful analytics tools in minutes. Reusable code for neural networks and clustering algorithms can be found online, and LLMs can be used to quickly generate functional software. Even within organizations, advanced analytics capabilities are already embedded in software applications. Statistical platforms such as IBM SPSS Statistics and Minitab now include neural network modelling, clustering, and other machine-learning capabilities directly in their menus. In other words, AI is already present in the tools that many organizations use every day. Given this reality, trying to keep AI out of an organization is not a realistic strategy. A more effective approach is governance, education, and responsible adoption. Employees need guidance on when and how AI should be used, what data can and cannot be shared with external tools, and how to critically evaluate AI. Training also helps professionals understand the limitations of AI systems, including bias, data quality issues, and the importance of human judgment in decision-making. Organizations can also reduce risk by deploying internal large language models trained only on approved corporate data. These internal systems allow employees to benefit from AI while keeping sensitive information within controlled environments. AI is already part of the professional landscape. The question organizations face is whether employees will use AI responsibly. That responsibility begins with leadership, governance, and education. Rather than blocking access, organizations should focus on guiding responsible AI use. Note: This content is based on confidential interviews with project managers |
Posted on: April 20, 2026 08:00 AM
| Permalink |
Comments (0)
Why Business and Academia Often View AI Differently
| From working in both academic research and industry, one observation is that artificial intelligence (AI) is often viewed very differently in these two environments. In academic research for business, AI is rarely the starting point of the study. When submitting research papers to journals, reviewers typically expect the research to begin with a clearly defined project problem, such as improving forecasting accuracy, identifying risk patterns, or optimizing decision-making. The researcher then uses theory and historical studies to evaluate analytical approaches to address that problem, and AI may be one of the methods considered. AI-based methods must withstand the rigour of comparison with other techniques, such as regression analysis, statistical models, or optimization methods. In other words, the research question comes first and the analytical process comes second. In business, however, the mindset is often quite different. Executives tend to view AI less as one analytical method among many and more as a strategic opportunity. AI has the potential to improve productivity, automate complex tasks, analyze large volumes of data, and generate insights faster than traditional approaches. Because of this, many organizations feel pressure to adopt AI quickly to avoid falling behind competitors. In practice, this leads to a very different starting point. Rather than asking, “What is the best analytical method for this problem?” business leaders often begin with the question, “How can we use AI to improve results?” This difference does not mean one perspective is right and the other is wrong. Academia emphasizes rigour, comparison, and methodological clarity. Business emphasizes speed, opportunity, and competitive advantage. The most productive path forward may lie somewhere between the two. Organizations benefit when they adopt AI thoughtfully, understanding both its potential and its limitations. At the same time, researchers can ensure their work remains relevant by studying real organizational challenges where AI is being deployed. Artificial intelligence is both a powerful tool and a strategic capability. Bridging the gap between academic research and business urgency may ultimately lead to better decisions in both worlds. |
Posted on: April 13, 2026 08:00 AM
| Permalink |
Comments (5)
The Flaw with Human in the Loop
| Many organizations and AI frameworks use the phrase “human in the loop” to describe the interaction between individuals and AI. However, the formal definition of the phrase means being aware of current events. in the loop /ˌin T͟Hə ˈlo͞op/ idiom included in a group of people who are informed about or involved in something; aware of what is happening. — Oxford English Dictionary Being informed is not the same as having decision authority or responsibility for outcomes. When organizations implement AI systems, simply keeping a person “in the loop” may not provide the level of engagement necessary to ensure responsible decision-making. Instead, I encourage students and workshop participants to “collaborate” with AI. Collaboration implies working together to achieve an objective. If humans and AI processes work together, the outcome should improve, or at least be more fully understood in the context of making a decision. The distinction matters. Being in the loop suggests awareness. Collaboration requires involvement. If AI unknowingly implements a highly biased resource plan, is it enough for the project manager to be aware of it? A situation that requires corrective action necessitates a deeper level of understanding than merely being informed. Collaboration means understanding the process by setting the objective, developing a data collection plan, performing the analysis, and delivering an actionable output. Collaboration does not mean constantly monitoring AI. It means ensuring the process is designed for quality and robustness, not speed or productivity. AI-based algorithms can predict or classify, and they do so with an associated probability of accuracy. Being in the loop means you understand the concept. Collaborating means you are part of developing and perhaps approving the process. If you are simply in the loop and an AI-driven decision results in a significant financial loss, how responsible would you feel? Collaboration implies a different level of accountability. It requires understanding the risks and addressing them proactively, just as organizations would in any well-managed software deployment. Errors may still occur, but a collaborative approach makes them easier to detect, understand, and mitigate. Human oversight of AI is an ethical and governance necessity. How oversight is implemented will vary across organizations. Describing humans as “in the loop” may unintentionally suggest passive awareness when what is truly needed is more active involvement. The success of AI systems depends not just on technology but on how clearly we define the human role that governs them. |
Posted on: April 06, 2026 08:00 AM
| Permalink |
Comments (8)
