When using AI systems, what are some best practices for ensuring the results you receive are accurate, relevant, and aligned with your original goals?

I find it helpful to use multiple LLMs and have each compare the responses of the other or two models. Usually I can get consensus across the models quickly.

Evaluating accuracy, relevancy and alignment needs focus on differents aspects of using AI systems: For example

Accuracy - whether the AI system responses are backed by source citations or whether the prompt engineeing obligates the FM to quote them

Relevancy - what is the cutoff date on which model was trained. Was RAG or finetuning being used to pull most current data. In addition, does prompt engineeing make sure the context is clear.

Alignment to goals - In addition to having the necessary data, is the promt engineering precise to ensure alginment with goals

Evaluating accuracy, relevancy and alignment needs focus on differents aspects of using AI systems: For example

Accuracy - whether the AI system responses are backed by source citations or whether the prompt engineeing obligates the FM to quote them

Relevancy - what is the cutoff date on which model was trained. Was RAG or finetuning being used to pull most current data. In addition, does prompt engineeing make sure the context is clear.

Alignment to goals - In addition to having the necessary data, is the promt engineering precise to ensure alginment with goals

Yes indeed, as a project management practitioner, one of my greatest fears about using GenerativeAI Agents (Tools) in my workplace(s) is trusting the Large Language Model (LLM) that the GenerativeAi tool is pulling data from. I ask myself “Can I trust what it’s telling me?”

ROOM FOR OPTIMISM

I think that I can rest a bit easier.

Prompt engineering is an artform!

PMI’s “Talking to AI: Prompt Engineering for Project Managers” Workbook 2024 helped me to better understand that I can use a reliability check like asking the AI for references and sources for its content.

I also learned of strategies project management practitioners’ can use to validate AI responses, for example:

Start by establishing a verification system where outputs are consistently compared to trustworthy sources or assessed by experts in the field.

Additionally, I can consider deploying automated systems to match responses with authoritative databases.

Another effective approach is to conduct

controlled experiments using synthetic or anonymized data to evaluate how LLMs perform across various situations.

Lastly, promote regular feedback among teams working with Al to maintain and enhance the security and accuracy of these outputs.

We need to evaluate carefully.

I agree with the simplistic approach, I have had success when being overly clear in my prompting, avoiding complicated vocabulary or sentence structures.

Start with a clear goal. Provide high quality input. Verify and refine output.

As they say, the answer always depends on how you ask the question. With AI tools, to receive quality answers, it is necessary to identify the actor's role, provide a good description of the context, and clearly specify the format of the data you want to receive.

I used to follow iterative approach in my prompts. Also will redefine my prompt based on the previous outputs. Basically, being specific and format samples will help us to get almost the expected output, but iterative is just one step to get 100% expected result