Michael Rowe

Hallucination

note5 min read

A hallucination sounds exactly like a true answer

A language model hallucinates when it produces something false with the same fluency and confidence as something true: a reference that doesn’t exist, a guideline that says something it doesn’t, a drug interaction nobody has described. What makes it dangerous is that nothing in the output marks it as an error.

Hallucination

One-sentence definition: A hallucination is output from a language model that is presented as fact, reads as plausible, and is false or unsupported by any source.

Why AI hallucinates

A large language model doesn’t look things up. It generates text one piece at a time by predicting what is likely to come next, given its training and whatever is in its context. Most of the time the likely text is also the true text, because true statements are what the training data mostly contains. When the model reaches the edge of what it has seen, such as an obscure paper, a recent guideline, or a precise figure, it keeps producing likely-sounding text anyway, and the result can be a well-formed citation to an article that was never written.

Kalai et al. (2025) argue that training and evaluation make this worse. Most benchmarks1 score an answer as right or wrong and give nothing for “I don’t know”, so a model that guesses outscores one that admits uncertainty, in the same way a student gains marks by guessing on a multiple-choice paper with no penalty for wrong answers. Rates have fallen as models have improved and as they’ve been connected to search and retrieval, and for some kinds of output they’re likely to fall to zero.

The difference is whether there’s a ground truth to check against. A reference either exists or it doesn’t, so a model that can search a database can verify a citation before it gives it to you. Much of what a model produces has nothing like that behind it: an explanation of why a treatment works, feedback on a student’s reflective writing, a suggestion about what to prioritise in a care plan. For that kind of output, a process that generates likely text and rewards confident answers will keep producing confident guesses, and hallucination is something the architecture and the training encourage, and it’s unlikely to disappear.

Why the name is contested

“Hallucination” borrows a term for a perceptual disorder, and some prefer “fabrication” or “confabulation”, the second of which is closer to what human memory does when it fills a gap with a plausible detail. Hicks et al. (2024) go further and argue that the output is bullshit in the philosopher Harry Frankfurt’s sense: produced with indifference to whether it is true, not with an intention to deceive.

Whatever the name, it describes a specific failure. A misspelling or a formatting slip is also an error, but not all AI errors are hallucinations, and the difference matters because a misspelling announces itself while a hallucination doesn’t.

In clinical and educational settings

Kim et al. (2025) define a medical hallucination as any instance in which a model generates misleading medical content. Testing models against physician-annotated responses to real clinical cases, they found that asking a model to reason step by step, or connecting it to search, reduced hallucination rates, but that a non-trivial rate remained.

For students, the familiar case is the reference list: a nursing student asks for sources on falls prevention in older adults and gets five convincing citations, two of which don’t exist. The student who checks each one in the library database catches it. The one who judges them by how plausible they look doesn’t, because plausibility is exactly what the model is good at producing.

Spotting hallucinations appears in most AI literacy frameworks as part of critical evaluation, but checking only works where there’s something to check against. Where there isn’t, the student is left judging the output on how it reads, and fluency is noise when it comes to judging accuracy. A human tutor gets things wrong too, which is the comparison AI tutor accuracy in health professions education takes up.

Sources

  • Hicks, M. T., Humphries, J., & Slater, J. (2024). ChatGPT is bullshit. Ethics and Information Technology, 26(2), Article 38. https://doi.org/10.1007/s10676-024-09775-5
  • Kalai, A. T., Nachum, O., Vempala, S. S., & Zhang, E. (2025). Why language models hallucinate (arXiv:2509.04664). arXiv. https://doi.org/10.48550/arXiv.2509.04664
  • Kim, Y., Jeong, H., Chen, S., Li, S. S., Lu, M., Alhamoud, K., Mun, J., Grau, C., Jung, M., Gameiro, R., Fan, L., Park, E., Lin, T., Yoon, J., Yoon, W., Sap, M., Tsvetkov, Y., Liang, P., Xu, X., … Breazeal, C. (2025). Medical hallucinations in foundation models and their impact on healthcare (arXiv:2503.05777). arXiv. https://doi.org/10.48550/arXiv.2503.05777

Footnotes

  1. Benchmark: a standard set of test questions used to compare AI models, much like a common exam paper sat by every candidate so their scores can be ranked. Language model benchmark on Wikipedia. ↩

Continue reading

  • Large language models

    A plain explanation of large language models: what they are, how they are trained, where their limits come from, and what their ability to produce fluent text means for assessment in health professions education.

  • Similarities between AI and human thinking: What if we're the language models?

    Discussion of AI and human cognition usually dwells on the differences. Turn the question around and the terminology of language models describes human thinking surprisingly well: context windows, biased training data, tokenisation, temperature, hallucination, system prompts, and pattern matching all have human counterparts, some of which health professions education already has names for. The similarities say something about our own cognitive architecture, and the strength of our resistance to them says something about what our professional identities depend on.

  • Retrieval augmented generation

    A technique that improves a language model's answers by retrieving relevant passages from your own documents and adding them to the prompt, so the model answers from sources instead of from memory alone.