AI IQ

I recently performed a study to compare results from project management students and large language models (LLMs). The students comprised 12 project management students, some of whom had project experience while others had none. All students were registered in a post-graduate program in project management and had completed at least half of their project management degree requirements, which included a course in creating, verifying, and managing scope documents. The meager incentive offered was the possibility of adding my name as a reference on their resume.

The scope document was a nine-page scope statement from a project to deploy Microsoft Dynamics software into a non-profit organization. The document had a few modifications to anonymize the organization and remove other references to named entities. The project was subject to schedule delays, which also incurred cost overruns, so there was a high probability of inconsistency in the scope statement.

The students were instructed to time their review of the scope document and list any and all errors or omissions found in the document. I also submitted the same scope document to ChatGPT 3.5 with the instruction to list errors or omissions and appended the document. The scope document was also submitted to the PMI Infinity project agent with identical instructions.

ChatGPT listed 20 errors or omissions, which were reviewed and considered valid. ChatGPT’s response time was 15 seconds. For the students, the average time was 65 minutes. They discovered an average of 5 errors identical to those found by ChatGPT. The students also found an average of 1 error per student that ChatGPT did not list, a total of 6 unique errors. The student-listed errors were reviewed and considered valid.

PMI’s agent responded in 18 seconds and, instead of finding errors, presented a list of activities that a project manager could perform to find them. As expected, based on PMI Infinity’s configuration, the response provides information for a project process and does not respond to a request for specific project details. 

Analysis Table

Conclusions

My observations are as follows:

1. ChatGPT was 268 times faster than the average project management student
2. ChatGPT is far more accurate but still missed some items
3. Not all LLMs are configured with the same data or serve the same purpose

This is only a preliminary study. I have easier access to students than experienced project managers who want to participate. It can be expected that results from experienced project managers would be an improvement, but it is doubtful that they would achieve the result in 15 seconds. Further study is needed to determine if a similar pattern occurs with experienced project managers compared to ChatGPT, where the LLM provides more comprehensive results but not total accuracy. 

I commonly see people express their personal belief that AI can never replace project managers because humans have better soft skills. This is an opinion, and I am not aware of any research that supports this assumption. In fact, I see evidence to the contrary. 

1. Incompetent project managers 

A survey in 2019 showed that 64 percent of employees would rather work for a robot than their current manager. That reflects my own work experience, as half of the managers I worked for were ineffective at managing a team. They had their own agenda, took credit for the excellent work of others, berated employees, and were terrible communicators. People have an idealistic view of how humans are all great managers when the reality is somewhat different. AI can be a great coach, delivering a fair and unbiased performance evaluation. AI can communicate positive and negative news without being judgmental.

1. Human bias

Based on the research of Nobel prize winner Daniel Kahneman, humans are burdened by personal bias. In other words, we have baggage that influences our actions and behaviors.  AI has far less baggage, and the only bias comes from the data selected by humans to feed the AI algorithms. AI bias can be identified and removed. Is it as easy or possible to do with humans?

2. Empathy

This is the most significant factor and least understood. When a human communicates with another human, it is based on the sender’s perspective.  The bias or baggage of the sender influences the message. Communication created by the sender is based on their personality, background, and emotions of the moment. AI reverses this flawed process. An AI algorithm evaluates the recipient first. What is the recipient’s personality, background, and feelings? AI creates a message based on how best to communicate with the recipient.  That is a powerful soft skill.

There are amazing project managers who have excellent soft skills. However, people struggle to understand each other and to communicate effectively. Evidence of this is the large volume of training courses on this subject. Properly programmed AI software provides better analysis and more effective transactions with team members. The ideal solution is for project managers to collaborate with and learn from AI tools. Working with AI offers the best outcome, where project managers rapidly learn and implement the soft skills needed to manage projects.

Although there are numerous methods for assessing project risks, they are still based on calculating a probability and impact. This is a 1980s concept that needs a fresh perspective. When you look at the list of project risks for a completed project, they either happened (1) or did not happen (0).  This is proof that risks are binary. As the world moves rapidly to a data-driven approach that takes advantage of AI technology, there needs to be a new paradigm for managing project risks.  Using data and AI tools means identifying the exact conditions if the risk will occur or will not occur. In other words, the probability is either 100 percent or 0. 

Risks are Not Random Events

Risks are not random events. Games of chance are random. A winning lottery number is (hopefully) randomly generated. Project risks are based on factors in the internal and external environment, where if there is sufficient data and data analysis, a binary decision can be made for probability. This process is available now for infrastructure projects and is popular in the UK. Having people assign a probability to a risk is full of human bias, adding another subjective value to our project plans. AI software that is trained to analyze risks reduces or eliminates this bias.

For skeptics who think risks are random events, here is my analogy. There was a significant wind storm recently, and I was worried that a large tree in front of my home would fall over and create damage.  The winds were 60 mph (80 km/hr.).  The tree was swaying and bending severely with each gust.  However, the risk was binary.  Either the tree would survive, or it would break and fall. To determine the binary result, I could gather data on the exact type of tree, height, width, age, number of branches, and soil conditions for the roots.  Then, I collect data for similar trees and the results when faced with identical wind conditions.  That determines the binary risk of the tree surviving.  What about a situation where the probability indicates a 20 percent risk of the tree falling over? Is that a realistic probability or a lack of data. There is a new way to manage project risks in a digital world.  The data needs to be collected, and that will be an arduous task until there is a critical mass that generates statistically significant results.

Making Project Risks Binary

The first step to a binary risk plan is to collect risk data.  Historical data, the current project conditions, and the future project environment are three categories of data necessary for binary risk management.  The process starts by collecting historical data.

• The specific risk
• The project conditions
• The environmental conditions
• Whether the risk occurred or did not occur

The next step is to identify the risks in the current project and gather data about the project and environmental conditions.  Once the data is collected, a machine learning algorithm uses classification to analyze risks to determine if they will or will not occur.

Does the process have to be perfect?  For now, we only need to be better than the old processes.  Using regression analysis, it is always possible to incorrectly classify a risk.  However, with accurate predictions, risks either become part of the project scope, schedule, and budget baseline or are ignored.  There is an argument that we cannot predict the future.  Yet, we know if we go outside when it is raining, we get wet.  We make predictions all the time, even for events that have not happened to us before.  If we walk in front of a fast-moving vehicle, we will get injured.  We cannot predict the future perfectly, but we can use AI technology to be more accurate at making these predictions.

Welcome to the new world of project risk management, where we replace human bias with advanced statistical methods that use AI technology.

For an initial implementation of machine learning, roughly 60 to 80 percent of the time is spent managing data. Data is a vital input to the algorithm but can be fraught with problems. The project environment is in a similar situation. The data needs to be structured and accessible. Does your organization have a data strategy for project data in preparation for utilizing AI-based software?

Data Wrangling

Structured data is maintained in a standard format with clear data definitions and is easily accessible. The requirement to achieve this is known as data wrangling. There are differing perspectives on the steps to take. However, it starts with identifying the data available. Project data is located in various areas, such as a scope document, project schedule, budget, and risk management plan. The formats are different, so accessing the data is a challenge. The next step is to clean the data. For those involved in a data migration project, there are numerous possibilities to create messy data.

Table. Examples of Problems in Data Fields

Once the data is clean, there should be some judgment if additional data is required. For project management, the status report might fail to include whether a resolution was successful or a risk response was effective. Project managers resolve issues but may not document the results in a format that can be captured as data.

Feature Engineering

It is usually insufficient to simply access data and successfully provide that input to a machine learning algorithm. The data needs to be modified. For example, two data fields might have the same meaning, so only one is selected. Three data fields might contain data, but taking an average for each entry provides a reasonable solution rather than overinfluencing the result simply by having three data fields. Feature engineering identifies missing data that is crucial to include or eliminates a data field that has no causal correlation.

The good news for project managers is that data scientists are less likely to have the ability to understand project data than project managers. As project managers, we know project processes and terminology. The data decisions are more appropriate, assuming the project manager has a basic level of training about how to manage data. Data scientists are in high demand and command high compensation. By performing a portion of the functions of a data scientist, project managers can dramatically increase their value to the organization.

The critical path was used extensively in the 1960s to enhance the project methodology for the US space program. Project managers still look for that red line in MS Project to identify activities that, if one task slips, the project end date is delayed. This concept is in serious need of modernization. Using AI tools, the critical path becomes more meaningful. The red line only determines the longest sequence of tasks based on precedence factors. A new AI-based critical path incorporates additional criteria.

1. Are the resources applied to critical path activities adequate to complete all the tasks as scheduled? This analysis includes comparing the task requirements to the skills being applied by the allocated resources.
2. Have the risks on the critical path been properly evaluated and investigated? Risks are troublesome issues for a project manager. AI improves the analysis and ensures mitigation is applied to critical path activities.
3. Are there any constraints that will impact the critical path? The theory of constraints offers an interesting perspective on how any activity can be delayed based on bottleneck factors.
4. Are the precedence relationships accurate? As the project progresses, activity relationships may change. We incorporate the precedence concept to recover a schedule by crashing or fast-tracking activities. Based on data, AI reevaluates how activities are connected and the possibility of changes.
5. Are there activities not on the critical path that will inevitably cause a schedule delay? All of the above points apply to how the critical path might change over time.       

The critical path is a fundamental and valuable concept in project management.  With new technology, such as AI, it is time to rethink how the critical path is applied to projects. AI can process more data and provide faster analysis than a human project manager. AI tools assess all the factors in real-time and notify the project manager in advance of issues. This is an opportunity to update the critical path concept using new technology and increase our expectations that the red line provides more meaning to project managers.