In a previous blog titled, An Empirical Study Comparing ChatGPT to Project Managers, I described a scope document study comparing the results of ChatGPT to the ability of project management students. In my research, ChatGPT was faster at identifying errors and more accurate, but it was only about 80 percent accurate. The study also revealed that some errors in the document were not listed by the large language model (LLM).
One of my student groups at the business college in France performed a variation on my work and included it in their major assignment. They compared three LLMs by requesting them to identify errors in a scope document.
Observations
The first observation is that LLMs perform at different speeds, although all are very fast compared to humans. The next observation is that each LLM had a different result. The explanation for different results should be obvious. Since the input document was identical for each LLM, the different results are based on what data each LLM could access. The quality and quantity of data used to train machine learning models are significant factors in producing accurate and reliable results. Acquiring high-quality data is becoming a priority for organizations that train machine learning models.
I want to thank the students for allowing me to use their assignment content for my blog: Dalton Bent, Carlos Carlson, Allen Jomy, and Vishal Venkata Penjarla.
Their study is one more example of the significance of data for AI technology to be successful.
