Ask an AI chatbot a question and you will usually get a fluent, confident answer. Most of the time it is useful. Sometimes it is simply wrong — a made-up date, a book that doesn’t exist, a court case no judge ever decided — delivered in exactly the same assured tone.
These errors are widely called “hallucinations.” They are not rare bugs that will vanish with the next update. Researchers increasingly see them as a predictable side effect of how today’s language models are built and tested. Understanding why they happen makes them much easier to spot.
What counts as a hallucination
In everyday use, a hallucination is any answer where a chatbot states something false as if it were true. OpenAI defines them as “plausible but false statements generated by language models.” The trouble is that a false answer usually sounds just as confident as a true one, so the reader gets no built-in warning.
OpenAI gave a memorable example in a September 2025 research post. When researchers asked a widely used chatbot for the title of the PhD dissertation of one of the paper’s own authors, Adam Tauman Kalai, it produced three different answers. None were correct. Asked for his birthday, it gave three different dates — all wrong.
Researchers at the University of Oxford use a narrower term for one type of error: “confabulations,” which they define as arbitrary and incorrect answers. A telltale sign is that the model gives different answers to the same question when you ask it more than once.
Why chatbots make things up
They learn patterns, not a list of verified facts
Large language models are trained on huge amounts of text to predict what words are likely to come next. According to OpenAI, this pretraining stage shows the model only examples of fluent text, without labels marking each statement as true or false. That makes it hard for the model to tell valid statements from invalid ones.
OpenAI points out that spelling follows consistent patterns, so those errors fade as models grow. But “arbitrary low-frequency facts,” such as a pet’s birthday, can’t be predicted from patterns — and that is where hallucinations creep in.
Tests reward guessing
The second reason is about incentives. OpenAI’s researchers argue that “standard training and evaluation procedures reward guessing over acknowledging uncertainty.”
OpenAI compares it to a multiple-choice exam: “If you do not know the answer but take a wild guess, you might get lucky and be right. Leaving it blank guarantees a zero.” OpenAI makes the same point with birthdays: a model that guesses a date has a 1-in-365 chance of being right, while a model that says “I don’t know” is guaranteed zero points. Across thousands of test questions, the guesser climbs the leaderboard — even though it produces far more false statements.
The numbers behind the trade-off
OpenAI illustrated this with results from SimpleQA, a test of short factual questions, comparing two of its own models:
- gpt-5-thinking-mini: declined to answer 52% of the time, was correct 22% of the time and gave a wrong answer 26% of the time.
- OpenAI o4-mini: declined to answer just 1% of the time, was correct 24% of the time and gave a wrong answer 75% of the time.
On pure accuracy, the older o4-mini model looks slightly better. But it got there by guessing on almost everything, and its error rate was nearly three times higher. A leaderboard that only counts correct answers would rank it ahead.
OpenAI’s proposed fix is to change how models are scored: penalize confident errors more than expressions of uncertainty, and give partial credit when a model appropriately says it isn’t sure. The company argues this has to happen in the main accuracy-based evaluations, not just in a few specialized tests on the side.
A real-world warning from a New York courtroom
The risks are not theoretical. In the 2023 case Mata v. Avianca, lawyers in a personal injury suit filed a court brief, drafted with ChatGPT’s help, that cited legal cases that did not exist. When the court asked for copies, the lawyers submitted fabricated material after the chatbot assured them the cases were real.
Judge P. Kevin Castel of the U.S. District Court for the Southern District of New York fined the attorneys $5,000 in June 2023 and found they had acted in bad faith. The underlying lawsuit against the airline was dismissed.
The lesson applies far beyond law: a chatbot’s confidence tells you nothing about whether a citation, quote or statistic is real.
Can software catch hallucinations automatically?
Researchers are working on it. A 2024 study in Nature, led by Yarin Gal’s group at the University of Oxford, introduced a method called “semantic entropy.” The idea is straightforward:
- Ask the model the same question several times.
- Group the answers by meaning, so “Paris” and “It’s Paris” count as the same answer even though the wording differs.
- Measure how scattered those meanings are. If the model keeps giving answers that mean different things, it probably doesn’t know.
The team tested the approach on five datasets covering trivia, general knowledge, biomedical questions and simple math word problems, and on several different model families, without training it for each task. It outperformed earlier detection methods, scoring 0.79 on a standard measure where 1.0 would mean perfectly separating right answers from wrong ones and 0.5 would be no better than a coin flip. The researchers also applied it to paragraph-length biographies written by GPT-4 and were able to flag made-up claims within them.
That is promising, but the authors are clear about its limits: it detects only arbitrary errors that come from gaps in knowledge. It cannot catch systematic mistakes, such as a falsehood the model learned consistently from bad training data, or failures of reasoning.
How to catch a hallucination yourself
You don’t need a research lab to apply the same logic. These habits go a long way:
- Ask twice. Regenerate the answer or ask the same question in a new chat. If key details change — dates, names, numbers — treat all of them as unverified. This is the do-it-yourself version of the semantic entropy idea.
- Be most suspicious of specifics. Exact figures, quotes, citations, page numbers and obscure names are the kind of arbitrary, rarely repeated facts that OpenAI says are hardest for models to get right.
- Check that sources exist. If a chatbot cites a study, article or court case, look it up yourself. A link or title that looks real is not proof.
- Invite uncertainty. Tell the chatbot it is fine to say it doesn’t know, and ask it to separate what it is confident about from what it is guessing.
- Use tools that show their sources for factual questions, and click through to the original rather than trusting the summary.
- Match your checking to the stakes. Brainstorming gift ideas is low risk. Anything involving money, health, legal matters or something you will publish deserves a primary source.
The catch
None of this makes chatbots useless. They remain helpful for drafting, summarizing and explaining. But even OpenAI’s own research frames hallucinations as a structural issue tied to how models are trained and graded, not a bug that one update will erase. Better scoring methods could make models more willing to say “I don’t know,” and detection methods like semantic entropy can flag some errors. Neither guarantees that what you read is true. For now, the most reliable fact-checker is still you.