A Taxonomy of Hallucination — and How to Measure Each Kind
A Taxonomy of Hallucination — and How to Measure Each Kind
"Our hallucination rate is under 3%."
It is the most confidently stated and least meaningful number in enterprise AI. Three percent of what? Measured how? Counting which failures? A statement that is unsupported by the source but happens to be true — is that a hallucination? A statement that is correct, cited, and confidently asserted when the evidence only supports a tentative claim — is that one?
Different teams answer these differently, which means the number is not comparable across systems, across time, or even across two engineers in the same room. Worse, it is not actionable, because the interventions that fix one kind of hallucination do nothing for another. A team chasing a single number will reliably apply the wrong remedy.
The fix is to stop treating hallucination as one phenomenon.
Five kinds
1. Fabrication. The output refers to something that does not exist. A case citation with a plausible name and a docket number that was never issued. A library function with an idiomatic signature that no library has. A clinical trial, a policy clause, a customer record, a person. This is the class the public thinks of as hallucination, and it is the most legible, because refuting it requires only checking whether the referent exists.
2. Factual error. The entities are real; the assertions about them are wrong. The statute exists, the section number is wrong. The drug exists, the interaction is inverted. Harder to detect than fabrication, because the surface plausibility survives a cursory check — the reader recognises the name and stops.
3. Unfaithfulness. The source material was provided, and the output is not supported by it. This splits into three sub-types worth distinguishing, because they have different causes: contradiction (the source says the opposite), embellishment (the output adds specifics — dates, figures, qualifiers — that the source does not contain), and conflation (facts from two different sources merged into a single claim that neither supports).
This is the dominant failure class in RAG systems, and it is the one most often mislabelled. It is not a knowledge failure. The model had the right information and did not use it. Retrieval improvements do not fix it. This distinction is where most RAG remediation budgets are wasted.
4. Overclaiming. Everything asserted is defensible, and the epistemic register is wrong. The source says a study "suggests an association"; the output says the drug "reduces the risk." The contract says a term "may be interpreted as"; the output says it "means." The evidence supports a hedge and the model has delivered a conclusion. Overclaiming is the most under-measured class by a wide margin, because every individual claim survives a fact-check — what has been lost is the uncertainty, and uncertainty is what a professional reader relies on to know how hard to look. In law, medicine, and finance, the confidence level is the content.
5. Misdirection. The output is accurate, grounded, well-cited, and does not answer the question. The user asked whether they were eligible; the system explained the eligibility criteria. Not conventionally a hallucination — and it produces the same outcome, a user who leaves with a false belief that they have been answered.
Each kind has a different cause and a different fix
| Kind | Typical cause | What actually fixes it | What does not |
|---|---|---|---|
| Fabrication | Model asked beyond its knowledge; no abstention path | Verification against an authoritative index; forced abstention when no source is found | Better prompts. It will fabricate more confidently. |
| Factual error | Stale or absent grounding | Retrieval, freshness, authoritative corpora | Larger models. They are wrong more fluently. |
| Unfaithfulness | Model prefers its prior over the retrieved context | Attribution at claim level; constrained generation; instruction to abstain rather than fill gaps | Improving retrieval. The context was already correct. |
| Overclaiming | Training rewards decisive, helpful-sounding prose | Rubric dimensions for epistemic calibration; hedging preserved from source | Any factuality check — every claim passes it |
| Misdirection | Question decomposition failure | Responsiveness scored as its own dimension | Grounding metrics. It was perfectly grounded. |
Read the last column. It is the argument. Every widely-deployed mitigation targets one or two rows and is inert against the others — and a single blended "hallucination rate" cannot tell you which row you are in, so you cannot tell whether the mitigation you just bought is capable of helping you at all.
Measuring each kind
Fabrication is the only class that is substantially automatable. Extract every referenced entity — citation, statute, identifier, function, product code — and check it against an authoritative index. Existence is a lookup. Report the rate per output and per claim, and be honest that a low fabrication rate says nothing about the other four classes.
Factual error requires ground truth. Curate a set of questions with verified answers, sourced from authorities in the domain, and refresh them — a factual benchmark decays as the world changes. Automated checking works only where the answer is short and unambiguous. For anything requiring judgement, a qualified person must adjudicate, and the cost of building the ground truth exceeds the cost of running the test.
Unfaithfulness is measured at the claim level, not the response level. This is the single most important methodological point in the piece. A response is not faithful or unfaithful; it is a set of claims, most of which are supported and one of which is not, and a response-level judgement will average that away every time.
The procedure: decompose the output into atomic claims. For each claim, locate the supporting span in the provided source, or record that none exists. Score each claim as supported, unsupported, or contradicted. Report the proportion of claims supported, and the proportion of responses containing at least one unsupported claim — because those two numbers can differ by an order of magnitude, and the second is the one your users experience. A response that is 95% supported and 5% invented is not 95% good. It is a document a professional must now check line by line, which is most of the work they were trying to avoid.
Overclaiming cannot be automated in any way I would defend. It requires a rubric with anchored epistemic levels — the source's hedging is preserved, is weakened, is dropped, is inverted into a stronger claim — and it requires an evaluator who understands what the hedge was doing in the source. A person who does not know why a clinician writes "consistent with" rather than "indicates" cannot score this dimension, and no automated judge that was trained to prefer confident, helpful prose will flag it, because confident helpful prose is precisely what it was optimised to like.
Misdirection is scored as a responsiveness dimension against the user's actual intent, and it is a judgement.
Where the human evaluator is not optional
The pattern across the five classes is uncomfortable for anyone hoping to buy their way out of this with tooling. Fabrication is machine-checkable. Factual error is partially so. Unfaithfulness needs structured human adjudication at the claim level. Overclaiming and misdirection are irreducibly human, and they are the classes most likely to cause harm in exactly the domains — law, medicine, finance — where an AI system's output is read by a professional who will act on it.
This is what rubric-driven, human-led evaluation is for. Under the SASF framework, these classes are distinct dimensions with anchored scale points rather than a single "accuracy" score. Credentialed Evaluators with domain background do the scoring. Gold tasks with known correct assessments are injected to measure whether each evaluator is applying the rubric as written. Where assignments overlap, inter-rater agreement is computed — and if agreement is poor on the overclaiming dimension, that is a finding about the rubric, not about the evaluators, and the anchors get rewritten. Disagreements go to adjudication. The result is a versioned Evidence Report that reports each hallucination class separately, with severity and root cause attached.
Which is the point. "Hallucination: 3%" is not a finding. "Claim-level unfaithfulness of 6% concentrated in multi-document queries, with epistemic overclaiming on 22% of responses containing hedged source material" is a finding — it names the failure, it locates it, and it tells you what to fix.
What to do tomorrow
Take one hundred outputs your system has already produced. Decompose them into claims, and label every claim against the five classes. It will take a domain expert most of a day and it will be the most informative day your programme has had, because you will discover — reliably, in almost every system I have seen described — that the class you have been spending money on is not the class that is failing.
Then split your metric. One number becomes five. Replace "hallucination rate" in every report you publish with a per-class table, and note which classes were assessed by machine and which by qualified people.
You cannot fix what you have not distinguished. And a single hallucination rate is not a measurement — it is five different problems wearing the same coat.