Director, Learning Design & Development| PMIAsheville, NC, United States
Validating and checking outputs is critical when working with AI systems like Generative AI. Such validation approaches may include establishing clear criteria, implementing strong testing protocols, and continuous refinement.
In your experience with AI, what are some best practices for ensuring the results you receive are accurate, relevant, and aligned with your original goals?
In my opinion and according to this course, I found these steps more effective in my Projects and collaborating with AI in my subjective matters .
Use Structured Prompts: Frame your inputs using established formulas like CREATE (Character, Request, Example, Adjustment, Type, Execution) or RTF (Role, Task, Format) to ensure clarity.
Provide Clear Context: Supply specific background data, guidelines, and target audience details rather than relying on the AI to guess.
Adopt an Iterative Mindset: Treat the AI like an assistant or mentee. Break complex tasks down into smaller sections ("chunking"), review the outputs, and give continuous feedback to refine the results.
Maintain Human Oversight: Never accept outputs at face value. Apply the Human-in-the-Loop principle by fact-checking, validating data against trusted sources, and running expert scrutiny over the final results.
A very good question and also difficult to answer as well. However you have to go to the basics and say as far as you are concerned, how well are you versed with the subject at hand ?. There are facts which the AI will generate and if you can verify these facts the more reliable the generated response will be. The fewer the facts then it means that the Generative AI response is far from meeting your original goals. Then it becomes very critical that you review the accuracy , relevancy and the alignment of the response to your original need. Unfortunately there are no clearly defined metrics that one can use a model to evaluate an AI generated response. So from my personal experience I basically restrict AI to an area where i have sound knowledge of , else it becomes almost impossible to verify details generated by an AI if you venture into unchartered territory. However with long usage and exposure your confidence also tend to increase as well. The best practice and protocol to follow would be to consult subject matter expects to validate the AI generated response before making critical decisions based on it to avoid any inherent associated risks which you might be not aware of.
Always use the CREATE model of prompts for better results. reiterate the prompts accordingly for best results. The output should always be validated before publishing. Saving Changes...
Anonymous
Personally, I like to cross reference with our DAFIs and other guidance to ensure things are aligned. I triple check the AI tool and perform the iterative prompt refinement. I also know that context is everything and the as of data date within the tool matters as well. Saving Changes...
Anonymous
Including as part of my prompt a requirement for the AI to include its confidence level in response to me. I also ask AI to provide me with the currency of its information sources. Where I already have knowledge of the year of the most updated edition of a publication, I insist on AI giving me responses that are based on that precise edition.
In my experience, the best practice is to treat AI output as a well-prepared draft, not as a final decision.
In a retail technology and project delivery environment, AI can be very useful for preparing status updates, meeting summaries, risk registers, issue logs, communication drafts, and first-level analysis. However, these outputs can easily become misleading if the prompt does not include the right business context, project constraints, audience, or expected outcome.
For me, accuracy starts before the AI gives an answer. It starts with how we frame the prompt. A project manager should clearly state the role, objective, background, constraints, expected format, and success criteria. If the prompt is too general, the response may look polished but still miss the actual project reality.
The second practice is validation. AI output should be checked against trusted sources such as project records, approved decisions, meeting notes, system facts, vendor confirmations, and subject matter expert input. This is especially important when the response involves timelines, risks, cost impact, compliance, integrations, or operational readiness.
The third practice is confidentiality. In retail projects, we often deal with sensitive information such as sales data, vendor details, commercial terms, system architecture, customer information, and internal decisions. These should not be shared with AI tools unless the organization has proper governance and approval in place.
The fourth practice is refinement. The first AI response is rarely the best one. I usually see better results when the output is reviewed, gaps are identified, and the prompt is refined with more context or clearer constraints. This makes the response more relevant and aligned with the original goal.
Ultimately, AI can help project managers work faster and structure information better, but the project manager remains accountable for accuracy, relevance, confidentiality, and final judgment. Human review is not optional. It is part of responsible AI use. Saving Changes...
AI is a broader term. Generative AI is just an ancient model but everything "explode" when Google published the new architecture called transformer in 2017. So, with that said, take into account that generative AI is just "predictive test with steroids" just simplifying the model. With that said, two key points has to be taking into account when somebody works with AI: 1-human in the loop. 2-AI without Data (today called data science discipline or big data or whatever) is the same thing that live without oxygen. Talking about generative AI all related to technology has almost not impact with relation to all related to non-technological roles and activities. What you stated about accuracy and things like that are easy to implement because there are a lot inside disciplines like statistics. Most of them to make things "a priori" to prevent instead of cure. Few organizations taking into account that when generative AI environments are put in place almost a new business unit has to be created where roles like lawyers, linguistic, diversity and inclusion specialist must be hire to help on put it in place.
Oversite and validation is very important for effective use of AI. In the end it is a tool which works based on previously gone through data and scenarios.
By following CREATE a decent response can be expected.
Saving Changes...
SEEMA AGGRAWALLUBRICANTS ADVISOR| IMPERIAL OILNiagara Falls, Ontario, Canada
Jun 08, 2024 1:37 PM
Replying to Keith Novak
...
Like with any new tool, you need to test the results before you scale up.
Think about if you were to manually model a very complex problem in a spreadsheet. You don't build all the links and formulas first and then evaluate your final output. You build and test sections of the bigger solution first and then add on layers once you have validated the functionality.
It takes several attempts to get to the right flow of prompts. Otherwise I find the results start becoming vague as more details are added Saving Changes...
PMI Team Member
