Hallucination

A hallucination is model output that is fluent, confident and false. The term is imprecise — the model is not malfunctioning, it is doing exactly what it was built to do, which is produce a likely continuation rather than a true one — but it is the term in use, and OWASP catalogues the risk as LLM09:2025 Misinformation.

Why it is a security topic and not only a quality one

Because software acts on it.

• Package hallucination. A coding assistant invents a plausible dependency name. An attacker registers that name on the public registry, and everyone who follows the suggestion installs it. This is a supply-chain compromise created by an output-quality defect, and it is the clearest example of the category.
• Fabricated citations and authorities. Invented case law, standards clauses and CVE identifiers get quoted into decisions and documents, and are expensive to detect after the fact.
• Confident wrong answers in a control path. An assistant that decides eligibility, classifies a ticket, or answers "is this transaction normal?" produces the same fluent output whether it knows or not.

Why retrieval reduces it without eliminating it

Grounding a model in retrieved documents helps, considerably. It also introduces a new failure mode: the model summarises a retrieved document that is itself wrong, out of date, or planted, and the citation makes the answer more credible rather than less. Retrieval moves the trust question from the model to the corpus; it does not remove it.

What actually helps

• Require citations that resolve, and check that the cited source contains the claim rather than merely existing.
• Constrain output to a schema or an enumerated set wherever a downstream system will act on it.
• Verify identifiers against a real registry before use — package names, account numbers, product codes, CVE IDs.
• Keep a human in the loop for anything irreversible.
• Tell users, in the interface, what the system is unreliable at. A general disclaimer at the bottom of a page is not that.

The dependable framing: a model's output is a proposal, and every proposal that reaches a system of record needs something deterministic between it and the record.