Lessons0 of 20 done

Module 2: Language models. Lesson 7 of 20, 7 minutes.

Why models make things up.

Why a model can state something false with total fluency, where that happens most, and the habits that catch it before it reaches your work.

In a widely reported case, a lawyer used a chatbot to help with legal research and filed a court brief citing several past decisions. The citations looked right: case names, courts, page numbers, quoted passages. None of the cases existed. The judge could not find them because there was nothing to find.

The lawyer had not skipped the obvious step. He had asked the chatbot whether the cases were real, and it assured him they were. This lesson is about why that happens, when to expect it, and the handful of habits that catch it.

A fluent answer that is not true

The word people use for this is hallucination: the model produces a confident, well-written answer that is false. It might be a wrong date, a book that was never written, a statistic with no origin, a quote nobody said, or a feature of a software product that does not exist. The defining feature is that nothing in the text signals the problem. The invented case citation is formatted exactly like a real one. The made-up quote sounds exactly like something the person would have said.

That is what makes it different from a typo or a visible gap. When a search engine finds nothing, it says so. When a model has nothing, it often writes something anyway.

Plausible is not the same as true

Recall what the model was trained to do: predict the next token, the next piece of a word, so that the text continues the way text usually continues. It became extremely good at producing text that fits. When you ask for the year a book was published, the model produces a year that fits: four digits, in the right range, in the right spot in the sentence. If the real year appears often in its training text, the fitting year and the true year are the same. If it does not, the model still produces a fitting year, because fitting is the only thing it knows how to do.

There is no separate step where the model checks the answer against a store of facts. It has no such store. What it knows is spread across billions of numbers as patterns and tendencies, not as a list it can look things up in. So the model cannot easily tell the difference between "I know this" and "this is the kind of thing that would be said here." From the inside, both are the same operation.

This also explains the second failure in the lawyer's story. Asked whether the cases were real, the model produced the text that usually follows such a question in a confident assistant's reply: yes, they are real. Checking would have required a tool. Answering only required a pattern.

Confidence in tone is not confidence in fact

People read tone as a signal. A hesitant answer makes us check, a crisp one makes us trust. That instinct works reasonably well with other people and fails with models. The training that made the model a helpful assistant (the fine-tuning and human feedback from the last lesson) rewarded clear, direct, well-organized answers. A model that hedged every sentence was rated lower. So the assistant learned a confident register, and it uses that register whether the underlying pattern is strong or weak.

Models have gotten better at saying "I am not sure," and you should take that seriously when you see it. But its absence tells you very little. A smooth paragraph with no hedging is the default style, not a measure of certainty.

When it happens most

Hallucination is not evenly spread. It clusters where the answer needs to be exact and the pattern is thin. Watch for it with:

  • Precise facts and numbers. Populations, percentages, measurements, prices. The model knows the shape of the number better than the number.
  • Dates. Especially for anything that is not famous.
  • Citations and references. Paper titles, authors, journals, web addresses, case law. These have a rigid format the model has seen millions of times, so inventing a convincing one is easy.
  • Quotes. Asked for a quote, the model produces what the person plausibly would have said, in their voice.
  • Obscure topics. A small town, a niche hobby, a minor historical figure, your own company. Little training text means thin patterns.
  • Anything after its knowledge cutoff. If the app has no search feature, the model has no information about recent events and may answer anyway.

Ask about the plot of a famous novel and you will almost always get it right. Ask for the page number of one particular line and you are in dangerous territory.

How to catch it

You cannot stop a model from hallucinating, but you can make it much harder for a hallucination to reach your work.

  1. Ask for sources, then open them. Not "does this have a source" but Give me the web address or full reference for each claim. Then click. A reference you cannot find does not exist until proven otherwise.
  2. Make it quote the material you gave it. If you pasted a document, ask Quote the exact sentence from the text above that supports this. A model is far more reliable when it is reading than when it is remembering.
  3. Ask how sure it is and what would change its answer. How confident are you in that, and what would make you wrong? This does not give you a true probability, but it often surfaces the weak link.
  4. Cross-check. Ask a second model, or run a plain web search. Two independent sources agreeing is worth far more than one source sounding sure.
  5. Prefer tools that can search. When the app can search the web or read your files, the model answers from text in front of it instead of from memory. That cuts invention sharply, though it does not remove it.
  6. Treat invented references as normal. Not a glitch, not a sign of a bad product. It is a known behavior of the technology, and the people who use it well simply plan for it.

Try it

Pick a real but not famous book or article you know well, something only a small audience would have read. Ask a chatbot: Who wrote [title], when was it published, and can you give me two direct quotes from it? Check each part against the real thing. Then ask: How confident are you in each of those, and which would you want to verify? Notice whether the confident tone of the first answer survives the second question.

Key terms

Hallucination
A fluent, confident answer from a model that is not true.
Plausible
Text that fits the pattern of what usually comes next, whether or not it happens to be true.
Citation
A reference to a specific source, such as a paper, book, web page or court case, meant to let someone check a claim.
Cross-check
Confirming an answer against an independent source, such as another model, a search or the original document.

Check yourself

1.Why does a model sometimes invent a book or a court case?
2.A chatbot gives you a confident, well-organized paragraph with no hedging. What does the tone tell you about accuracy?
3.Which of these questions carries the highest risk of hallucination?

A model writes what fits, not what has been checked, so treat every precise fact, number, date and reference as unverified until you have opened the source.