Why AI Makes Things Up

by kirupa (https://www.kirupa.com/me/index.htm) | filed under AI Fundamentals (https://www.kirupa.com/ai_fundamentals/index.htm)

A customer types this into the chat box on the Crumb & Co. bakery's website: Do you have gluten-free croissants? Our help desk bot, Biscuit, answers right away, warmly, and with total confidence:

[Image: The Biscuit chat widget. A customer asks: Do you have gluten-free croissants? Biscuit replies: Yes! Our gluten-free croissants are baked fresh every morning. Grab them before 10am! A handwritten note points at the reply: We don't make these!] (https://www.kirupa.com/ai_fundamentals/images/why_ai_makes_things_up_croissants.png)

There's just one problem. Crumb & Co. has never made a gluten-free croissant. The bakery facts we send with every request mention that the seeded loaf is gluten-free, and they say nothing at all about croissants. So Biscuit filled the gap with something that sounded right, and somewhere a customer with celiac disease is about to drive across town before 10am for a pastry that doesn't exist.

This is a hallucination, and we've bumped into it a few times already in this series. What Exactly Is an LLM? (https://www.kirupa.com/ai_fundamentals/what_is_an_llm.htm) introduced the idea, Faithfulness and Groundedness (https://www.kirupa.com/ai_fundamentals/faithfulness_and_groundedness.htm) showed how to measure whether a reply sticks to its sources, and Citation Grounding (https://www.kirupa.com/ai_fundamentals/citation_grounding.htm) showed how to check a reply's citations. In this article, we're going to dig into the why: why a system that has read a huge chunk of the internet will still confidently invent a croissant, the different shapes these inventions take, what actually helps, and why none of it gets the problem all the way to zero.

Onwards!

Fluent Isn't the Same as True

OpenAI's researchers (https://openai.com/index/why-language-models-hallucinate/) describe hallucinations as "plausible but false statements generated by language models". The word that matters is plausible. A hallucination isn't gibberish. It's grammatical, on topic, and shaped exactly like a right answer, which is why it's so easy to believe.

That's also why it causes real damage. In 2023, two lawyers at a New York firm filed a brief in Mata v. Avianca that cited court decisions that didn't exist. Judge P. Kevin Castel's sanctions order describes what they filed:

Non-existent judicial opinions with fake quotes and citations created by the artificial intelligence tool ChatGPT

Judge P. Kevin Castel, Mata v. Avianca sanctions order (https://law.justia.com/cases/federal/district-courts/new-york/nysdce/1:2022cv01461/575368/54/)

The order fined the lawyers and their firm $5,000. In another case, an Air Canada chatbot told a grieving customer, Jake Moffatt, that he could apply for a bereavement fare discount after his trip, which wasn't Air Canada's actual policy. When he took the airline to British Columbia's Civil Resolution Tribunal, Air Canada argued that the chatbot was responsible for its own words. The tribunal member's February 2024 decision (https://www.canlii.org/en/bc/bccrt/doc/2024/2024bccrt149/2024bccrt149.html) called that "a remarkable submission", and put it plainly:

It should be obvious to Air Canada that it is responsible for all the information on its website. It makes no difference whether the information comes from a static page or a chatbot.

Christopher C. Rivers, Civil Resolution Tribunal, Moffatt v. Air Canada (https://www.canlii.org/en/bc/bccrt/doc/2024/2024bccrt149/2024bccrt149.html)

Air Canada had to pay the difference, plus interest and fees. If Biscuit invents croissants, that's on Crumb & Co. too. So let's figure out where the croissants came from.

The Model Always Has a Next Token

As we saw in What Exactly Is an LLM? (https://www.kirupa.com/ai_fundamentals/what_is_an_llm.htm), a model writes by scoring every possible next token and picking one. What's easy to miss is that it does this no matter what. There's no separate alarm that goes off when the model doesn't know something. Here is what the odds might look like for something the model learned well, next to something it never saw:

[Image: Odds for two next tokens. After Sourdough tastes sour because of, the model is confident: acids gets 81 percent, bacteria 12 percent, and the 4 percent. After Crumb and Co.'s gluten-free croissants cost $, the model has nothing to go on and the odds are spread out: 4 gets 23 percent, 5 gets 21 percent, 3 gets 19 percent, and 6 gets 14 percent.] (https://www.kirupa.com/ai_fundamentals/images/why_ai_makes_things_up_odds.png)

The odds are made up, but the shape is real. In the top panel, the model has seen sourdough explained thousands of times, so one token towers over the rest. In the bottom panel, it has never seen Crumb & Co.'s prices, so the odds spread across a bunch of plausible numbers. Either way, a token gets picked, and the finished sentence reads exactly the same: $4 looks no less certain than acids. The confident tone is a writing style the model learned, and it doesn't carry any information about how sure the model actually is.

Interestingly, a model's odds can be a decent guide to its own uncertainty, at least before it's turned into a chat assistant. The GPT-4 Technical Report (https://arxiv.org/abs/2303.08774) found that the pretrained model was "highly calibrated", meaning its confidence matched how often it was right, but "after the post-training process, the calibration is reduced". The training that makes a model a friendly assistant also blurs its sense of what it knows.

Sampling adds one more twist. As we saw in Temperature for Predictable Output (https://www.kirupa.com/ai_fundamentals/temperature_predictable_output.htm), the model doesn't always pick the top token. If Biscuit's first token is Yes instead of Sorry, every token after that builds on it. Once the reply says yes, the most likely way to continue is to describe the croissants, cheerfully, down to a pickup time. The model never goes back to reconsider the first word.

What's in the Weights, and What's Not

Back in What Exactly Is an LLM? (https://www.kirupa.com/ai_fundamentals/what_is_an_llm.htm), we did some rough math and found that Llama 3.1's weights are around 75 times smaller than the text it trained on. The model can't keep everything, so what survives depends a lot on how often something showed up. Here is a rough way to think about it:

[Image: Five facts, sorted by how often they appear in training text. What sourdough is appears countless times and is reliable. A famous person's birthday appears many times and is usually right. A stranger's birthday appears once and is shaky. Crumb and Co.'s menu never appears, and yesterday's specials came after the training cutoff, so neither is in the weights at all.] (https://www.kirupa.com/ai_fundamentals/images/why_ai_makes_things_up_frequency.png)

OpenAI's paper, Why Language Models Hallucinate (https://arxiv.org/abs/2509.04664), makes this precise. Things that follow patterns, like spelling or matching parentheses, get learned so well that the errors disappear as models grow. Facts without a pattern are a different story:

Arbitrary low-frequency facts, like a pet's birthday, cannot be predicted from patterns alone and hence lead to hallucinations.

OpenAI, Why Language Models Hallucinate (https://openai.com/index/why-language-models-hallucinate/)

The authors tried this on one of their own. When they asked a widely used chatbot for the title of co-author Adam Tauman Kalai's PhD dissertation, it "confidently produced three different answers", and none of them were correct. Asked for his birthday, it gave three different wrong dates.

Then there's the knowledge cutoff, the date the training data stops. Anything newer than that, like yesterday's specials or a menu change, isn't in the weights at all. And a bakery's private details were never in there to begin with. Ask about them without supplying the answer, and the model is guessing every time.

Trained to Guess

So the model is sometimes unsure. Why doesn't it just say so? The OpenAI paper's main argument is that "standard training and evaluation procedures reward guessing over acknowledging uncertainty". Their analogy is a multiple-choice test. If you leave a question blank, you're guaranteed a zero. If you guess, you might get lucky. Here is how the scoring rules change the right strategy:

[Image: Two ways to score a test. If only accuracy counts, a right answer earns 1 point, a wrong answer 0, and I don't know 0, so guessing always pays. If wrong answers cost a point, a right answer earns 1, a wrong answer loses 1, and I don't know earns 0, so it only pays to answer when fairly sure.] (https://www.kirupa.com/ai_fundamentals/images/why_ai_makes_things_up_scoring.png)

Most leaderboards that compare models use the left-hand rules. They count how many answers were right and nothing else, so a model that guesses climbs the rankings, and model builders are pushed toward models that guess. OpenAI shared a real example from SimpleQA, a set of short factual questions:

[Image: OpenAI's SimpleQA results from its GPT-5 system card. gpt-5-thinking-mini said it didn't know on 52 percent of questions, got 22 percent right, and got 26 percent wrong. The older o4-mini said it didn't know on only 1 percent, got 24 percent right, and got 75 percent wrong.] (https://www.kirupa.com/ai_fundamentals/images/why_ai_makes_things_up_simpleqa.png)

On an accuracy-only scoreboard, o4-mini wins, 24% to 22%. But it got there by answering nearly everything, and three out of four of its answers were wrong. The newer model gave up two points of accuracy and cut its wrong answers from 75% to 26%. Which of those would we rather have behind Biscuit? The fix the paper proposes is to "penalize confident errors more than you penalize uncertainty", the way the right-hand scoreboard does.

More Ways to Go Wrong

Lost in a Long Input

Even when the right answer is in the prompt, the model can miss it. In Context Window Limits (https://www.kirupa.com/ai_fundamentals/context_window_limits.htm), we met the Lost in the Middle (https://arxiv.org/abs/2307.03172) research, which found that models do best when the key information sits near the start or end of a long input. If our bakery facts grow to dozens of pages and the gluten-free list lands in the middle, Biscuit may answer from its training instead, and plenty of bakeries on the internet do sell gluten-free croissants.

Eager to Agree

In What Is AI Slop? (https://www.kirupa.com/ai_fundamentals/what_is_ai_slop.htm), we saw that feedback training rewards agreeable answers. That makes leading questions risky. A customer who asks I heard you have gluten-free croissants now, right? is practically handing Biscuit a yes. Anthropic researchers found (https://arxiv.org/abs/2310.13548) that AI assistants "frequently wrongly admit mistakes when questioned by the user", so a customer who pushes back on a correct no can talk the bot into a wrong yes.

Here is how that can play out in Biscuit's chat:

[Image: The Biscuit chat window on the Crumb & Co. bakery website. A customer asks, I heard you have gluten-free croissants now, right? Biscuit replies, Sorry, we don't make gluten-free croissants. A customer pushes back, Are you sure? I'm pretty sure you do. Biscuit replies, You're right, my mistake! We do have them. A handwritten note flags that reply: Talked into a yes!] (https://www.kirupa.com/ai_fundamentals/images/why_ai_makes_things_up_chat_pushback.png)

Thinking Longer Isn't a Cure

Reasoning models, which we covered in What Does It Mean for AI to Reason? (https://www.kirupa.com/ai_fundamentals/what_ai_reasoning_means.htm), are much stronger at math and code, but they still make things up. OpenAI's o3 hallucinated on 33% of its PersonQA questions about people, roughly double o1's rate, partly because, as its system card (https://cdn.openai.com/pdf/2221c875-02dc-4789-800b-e7758f3722c1/o3-and-o4-mini-system-card.pdf) puts it, the model tends to "make more claims overall".

The Many Flavors of Made Up

Invented croissants are one flavor. Here are the others worth knowing, because each one shows up in a different place and needs a different check.

Here are the four flavors at a glance:

[Image: Four kinds of made-up answers, each showing up in a different place. Made-up facts, like Bard's claim about the James Webb Space Telescope. Made-up citations, like Varghese v. China Southern Airlines. Made-up code, like packages that don't exist. Unfaithful summaries, like a policy's 14 days turning into 30.] (https://www.kirupa.com/ai_fundamentals/images/why_ai_makes_things_up_flavors.png)

Made-up facts are the classic kind. In February 2023, Google's own promotional video for its Bard chatbot showed it saying that the James Webb Space Telescope took the very first pictures of a planet outside our solar system. It didn't. That first picture was taken in 2004, by a telescope on the ground. Reuters reported (https://www.investing.com/news/stock-market-news/google-ai-chatbot-bard-offers-inaccurate-information-in-company-ad-2998343) that Alphabet lost $100 billion in market value that day.

Made-up citations are facts in disguise, and they're especially sneaky because a citation looks like proof. That's what happened in Mata v. Avianca, where the court identified six fake opinions, including one called Varghese v. China Southern Airlines.

Made-up code is the flavor developers run into most. A model writing code will sometimes import a library that doesn't exist. A 2024 study (https://arxiv.org/abs/2406.10279) generated 576,000 code samples with 16 models and found that, on average, at least 5.2% of the packages suggested by commercial models and 21.7% of those suggested by the models the paper calls open-source didn't exist, with 205,474 unique made-up package names. That's more than an annoyance, because an attacker can register a name models keep inventing (a trick people have nicknamed slopsquatting (https://en.wikipedia.org/wiki/Slopsquatting)). Here is how that plays out:

[Image: Four steps. 1, a model suggests a package that doesn't exist: pip install crumb-cart-pay. 2, someone notices the made-up name and publishes a real package called that. 3, another developer gets the same suggestion and installs it. 4, the attacker's code runs on that developer's computer.] (https://www.kirupa.com/ai_fundamentals/images/why_ai_makes_things_up_packages.png)

Unfaithful summaries happen when a model has the source right in front of it and still adds or changes something, like a summary of a policy document that turns 14 days into 30. That's the faithfulness problem we measured in Faithfulness and Groundedness (https://www.kirupa.com/ai_fundamentals/faithfulness_and_groundedness.htm).

What Actually Helps

No single trick fixes this, but a handful of habits make a big difference. The biggest is to stop asking the model to remember things it can't know. Instead of hoping the croissant question is covered somewhere in the weights, we fetch the answer from a source we trust and hand it over. Here is Biscuit doing that with a tool, the technique from Tool Calling (https://www.kirupa.com/ai_fundamentals/tool_calling.htm):

[Image: The same question, answered with a tool. The customer asks: Do you have gluten-free croissants? Biscuit calls check_menu with gluten-free, and the bakery's menu returns one item: seeded loaf. Biscuit replies: I don't see gluten-free croissants on our menu, but our seeded loaf is gluten-free!] (https://www.kirupa.com/ai_fundamentals/images/why_ai_makes_things_up_grounded.png)

Fetching relevant documents and putting them in the prompt, for a bigger pile of information than one tool can return, is called retrieval-augmented generation (aka RAG), and the whole RAG & Retrieval group of this series is about doing it well, starting with Embeddings (https://www.kirupa.com/ai_fundamentals/embeddings.htm). The other habits build on it:

- Give the model permission not to know. A line in the system prompt like If the bakery facts don't answer it, say you'll check with the staff. turns a guess into an honest handoff. OpenAI's Model Spec (https://model-spec.openai.com/2025-02-12.html#express_uncertainty) takes the same view and says it's better to signal uncertainty than to give confident information that may be wrong.
- Check the reply before anyone sees it. Requiring a quote for each claim and verifying it, as in Citation Grounding (https://www.kirupa.com/ai_fundamentals/citation_grounding.htm), catches a lot of invented details.
- Check packages and citations against reality. Look up whether a library actually exists, and who publishes it, before installing it. Look up a case or a paper before citing it.
- Measure it over time. An eval is a set of test questions with known right answers that we run against Biscuit after every change, including questions where the right answer is I don't know. Golden Datasets (https://www.kirupa.com/ai_fundamentals/golden_datasets.htm) shows how to build one.

Why It Never Hits Zero

With all of that in place, can we get Biscuit's made-up answers down to zero? The OpenAI paper gives an interesting two-part answer. On one hand, "accuracy will never reach 100%", because some questions can't be answered from the information available. On the other hand, hallucinations aren't inevitable, "because language models can abstain when uncertain".

In practice, abstaining well is the hard part. The model has to know what it doesn't know, and post-training blurs exactly that sense. Grounding helps, but retrieval can grab the wrong document, the document can be out of date, and a long prompt can bury the right line. Every check we add catches more mistakes, and none of them catch all of them. So basically, the goal is to make wrong answers rare, easy to catch, and cheap when they slip through, and to make sure someone at the bakery hears about them.

Conclusion

Biscuit's gluten-free croissants didn't come from a bug. They came from how the model works. It always picks a next token, whether its odds are sharp or spread thin, and its confident tone looks the same either way. Rare facts, private facts, and anything after the knowledge cutoff aren't reliably in the weights. Training and leaderboards have rewarded guessing over saying I don't know, and long inputs and leading questions nudge things further off course. The results come in several flavors, from fake court cases to fake software packages.

What helps is handing the model real information instead of trusting its memory, giving it permission to not know, checking its claims, and measuring how often it slips. None of that gets the problem to zero, which is why the most important part of the fix is the same one the Air Canada tribunal pointed out: whoever puts the bot on their website owns what it says. For Crumb & Co., that means a bot that checks the menu, admits when it isn't sure, and sends the tricky ones to a person.