Ever chatted with an AI and gotten a response so out there, it felt like it came from a parallel universe?
LLMs (like all of us) occasionally have day-dreams, also known as "hallucinations".
Stick with me, and I'll show you a quick and easy fix to prevent this from happening to the product you're working on.
Different Types of Hallucinations:
The Obvious Untruths:
An AI just told you about a president who never existed.
The LLM has made up events, people, and places that are as real as unicorns, entertaining, but not exactly helpful when you're after the facts.
The Oversharer:
In contrast to making things up, some LLM responses go overboard with details. Before you know it, a user gets access to information it shouldn't have. It raises concerns about privacy and the ethical use of data.
The Gibberish Generator:
And sometimes, it's like they just smash the keyboard and hope for the best, similar to a baby with alphabet soup. Not what you want if this output is fed to another function.
Just a quick word about Pezzo: It is the fastest growing open source LLMOps platform, and the only one built for full stack developers with first class TypeScript Support.
Like our mission? Check Pezzo out and give us a star. We're building for developers, by developers 🧙♂️
Why Do Hallucinations Happen?
The Desire to Please:
LLMs hate letting us down. When faced with gaps in data, it may invent details to fulfill the user's request, prioritizing continuity over accuracy.
Outdated Information:
An LLM's knowledge can be outdated. In fact, it is only as current as its last update. This can lead to hiccups in their responses.
Preventing Hallucinations With "Sanity Checks"
Sanity Checks are our way of teaching AIs to pause and ponder before blurting out whatever comes to mind. It's like giving them an internal filter to weed out the wacky before it hits our screens.
This is how you do it: Add to your prompt an example of a thinking process where you stop after every response and perform a check about that response. The subject doesn't matter, the LLM will follow the reasoning structure. You can copy and paste this example to your prompt:
Question: Who lived longer, Albert Einstein or Isaac Newton?
Answer:
Are follow up questions needed here: Yes.
Follow up: How old was Albert Einstein when he died?
Intermediate answer: Albert Einstein was 76 years old when he died.
Follow up: How old was Isaac Newton when he died?
Intermediate answer: Isaac Newton was 84 years old when he died.
So the final answer is: Isaac Newton.
The Evolution of Prompting
This prompting technique is a natural evolution from Direct Prompting and Chain-Of-Thought Prompting (see pic below). Here's the cool part: Sanity Checks Prompting reveals the LLM's thought process, letting us peek into the "black box" as it breaks down complex queries into bite-size follow-ups. The LLM knows it's reached the destination when it can seamlessly transition from intermediate answers to a final, confident conclusion.
AI hallucinations are a fixable glitch, not a dead end, and you're now ready to tackle them head-on.
Take these tools, push the AI to its limits, and make it work for you.
If there's a specific bug you can't iron out with your AI, let's hear it below. Your challenge might just be the star of our next troubleshooting guide.
Go on, give it a whirl and report back 🚀
This article is inspired by the paper Measuring and Narrowing the Compositionality Gap in Language Models
Top comments (0)