An Empirical Study Comparing ChatGPT to Project Managers

 

I recently performed a study to compare results from project management students and large language models (LLMs). The students comprised 12 project management students, some of whom had project experience while others had none. All students were registered in a post-graduate program in project management and had completed at least half of their project management degree requirements, which included a course in creating, verifying, and managing scope documents. The meager incentive offered was the possibility of adding my name as a reference on their resume.

The scope document was a nine-page scope statement from a project to deploy Microsoft Dynamics software into a non-profit organization. The document had a few modifications to anonymize the organization and remove other references to named entities. The project was subject to schedule delays, which also incurred cost overruns, so there was a high probability of inconsistency in the scope statement.

The students were instructed to time their review of the scope document and list any and all errors or omissions found in the document. I also submitted the same scope document to ChatGPT 3.5 with the instruction to list errors or omissions and appended the document. The scope document was also submitted to the PMI Infinity project agent with identical instructions.

ChatGPT listed 20 errors or omissions, which were reviewed and considered valid. ChatGPT’s response time was 15 seconds. For the students, the average time was 65 minutes. They discovered an average of 5 errors identical to those found by ChatGPT. The students also found an average of 1 error per student that ChatGPT did not list, a total of 6 unique errors. The student-listed errors were reviewed and considered valid.

PMI’s agent responded in 18 seconds and, instead of finding errors, presented a list of activities that a project manager could perform to find them. As expected, based on PMI Infinity’s configuration, the response provides information for a project process and does not respond to a request for specific project details. 

Analysis Table

Conclusions

My observations are as follows:

1. ChatGPT was 268 times faster than the average project management student
2. ChatGPT is far more accurate but still missed some items
3. Not all LLMs are configured with the same data or serve the same purpose

This is only a preliminary study. I have easier access to students than experienced project managers who want to participate. It can be expected that results from experienced project managers would be an improvement, but it is doubtful that they would achieve the result in 15 seconds. Further study is needed to determine if a similar pattern occurs with experienced project managers compared to ChatGPT, where the LLM provides more comprehensive results but not total accuracy.