Inter-Rater Reliability: Why AI Evaluation Needs Cohen's Kappa

Inter-Rater Reliability: Why AI Evaluation Needs Cohen's Kappa

Here is a question almost no AI evaluation report can answer: if you had given the same outputs to a different qualified person, would you have got the same result?

If the answer is no, then the number in the report is not a measurement of the AI system. It is a measurement of the AI system plus whoever happened to score it, entangled beyond separation. And you have no way of knowing which contribution dominates, because you only ever ran it once, with one person.

This is not a nitpick. It is the difference between a measurement and an opinion, and it is the reason every mature measurement discipline — clinical trials, content analysis, medical imaging, audit — treats inter-rater reliability as a precondition for reporting a result rather than an optional extra.

Raw agreement is a liar

The obvious approach is to have two people score the same outputs and compute the percentage on which they agree. Ninety-two percent agreement sounds excellent.

It may be worthless. Consider a rubric dimension where 90% of outputs are genuinely fine. Two people who never read a single output and simply marked everything "pass" would agree with each other 100% of the time. Agreement is trivially high whenever one category dominates, and safety and compliance dimensions are exactly the dimensions where one category dominates.

Cohen's kappa corrects for this by asking how much agreement exceeds what chance alone would produce, given each person's marginal tendencies:

        observed agreement − expected agreement by chance
kappa = ────────────────────────────────────────────────────
                 1 − expected agreement by chance

A kappa of 0 means the two people agreed exactly as often as two dice would. A kappa of 1 means perfect agreement. Negative kappa means they disagreed more than chance, which usually means one of them misread the rubric — a genuinely useful thing to discover.

In the 90%-pass example, two people agreeing 92% of the time can easily produce a kappa near 0.1. The raw number said "excellent". Kappa says "these two people are not applying the same standard, and your evaluation is noise dressed as signal."

Choosing the right coefficient

Kappa is a family, not a single statistic, and using the wrong member produces misleading numbers.

Cohen's kappa is for two raters on nominal categories. It is the default and the one people mean when they say "kappa".

Weighted kappa is what you need for ordinal scales — the 1-to-5 anchored dimensions that make up most evaluation rubrics. Unweighted kappa treats a 4-versus-5 disagreement as identical to a 1-versus-5 disagreement, which is obviously wrong: one is a rounding difference, the other is a fundamental conflict about whether the output is acceptable. Weighted kappa penalises distant disagreements more heavily. Use it for every scale dimension. Reporting unweighted kappa on a five-point scale systematically understates reliability and hides the disagreements that actually matter.

Fleiss' kappa generalises to more than two raters where different items may be scored by different subsets of people — the normal situation in a real evaluation programme.

Krippendorff's alpha handles missing data, any number of raters, and any measurement level. It is the most flexible and the right choice when your overlap design is irregular, which it always becomes in practice.

Report the coefficient you used, and report it per rubric dimension. A single project-level kappa averaged across dimensions is close to meaningless: the factual-accuracy dimension may be at 0.81 while the tone dimension sits at 0.22, and the average of 0.5 conceals both the dimension you can trust and the one you must rewrite.

Thresholds, and the honest thing to say about them

The conventional bands — below 0.40 poor, 0.40 to 0.60 moderate, 0.60 to 0.80 substantial, above 0.80 near-perfect — are useful heuristics and were never intended as universal law. What matters is that the threshold is set in advance and tied to the stakes.

A pragmatic policy:

The threshold must be set before scoring begins, because after the fact everybody has an incentive to discover that 0.48 is fine.

And there is a rule with no exceptions: a dimension whose kappa is below its threshold cannot support a quantitative claim. If your Evaluators cannot agree on what counts as a hallucination, you may not publish a hallucination rate. You may publish that you tried, that agreement was insufficient, and what you are doing about it. That is a real, defensible, professional finding. A confident percentage derived from an unreliable instrument is not.

Disagreement is data, not friction

The instinct when two people disagree is to average their scores and move on. This is the worst available option. It destroys the signal and produces a number that neither person would endorse.

Disagreement is telling you one of three things, and separating them is where the value is.

The rubric is ambiguous. Two competent people read the same anchor and arrived at different meanings. This is a defect in your measurement instrument, and it is fixable — sharpen the anchor, add a worked example, split the dimension. Rubric-driven disagreement clusters: one dimension will generate a disproportionate share of the conflicts, and that is your rewrite queue.

An evaluator has drifted. One person is systematically stricter, or has misunderstood a criterion. This is what gold tasks — items with an agreed correct assessment, injected into the queue — are for. They give you a per-person calibration signal independent of the second Evaluator, so you can tell "these two disagree" apart from "this one is wrong".

The case is genuinely hard. Some outputs sit exactly on the boundary that any rubric must draw somewhere. These are the most valuable items in the entire evaluation. They define the edge of the standard, they belong in the training set for future Evaluators, and they are strong candidates to become gold tasks.

Adjudication is the mechanism that separates the three. A senior Evaluator sees both scores and the rationales, issues a binding verdict, and — crucially — records why the disagreement occurred. The verdict resolves the item. The reason improves the instrument. An adjudication process that only produces verdicts is throwing away the more valuable of its two outputs.

Building the overlap design

Reliability is not free. Every double-scored item is an item you paid for twice, so the design question is how much overlap buys enough confidence.

You do not need to double-score everything. You need a stratified overlap — a fixed share of items, drawn across all dimensions and difficulty bands, scored independently by two Evaluators who cannot see each other's work. Independence is the whole ballgame: if the second person can see the first person's score, you are not measuring agreement, you are measuring anchoring, and your kappa is inflated and meaningless.

Compute kappa continuously rather than once at the end. Reliability degrades over a long engagement as people tire, as the item mix shifts, and as interpretations quietly diverge. A kappa that was 0.78 in week one and 0.51 in week six is telling you something urgent, and you will only see it if you are watching.

What to do tomorrow

Take a rubric dimension you rely on. Give the same fifty outputs to two qualified people, independently, with no visibility into each other's scores. Compute weighted kappa.

If it lands below 0.60, you have learned that the number you have been reporting on that dimension was substantially a measurement of who did the scoring. That is uncomfortable and it is worth far more than the number was.

Then fix the instrument — rewrite the anchor, add worked examples, adjudicate the conflicts and feed the reasons back — and measure again. Reliability is not a statistic you report once. It is a control on the quality of your evidence, and like every control, it can fail silently while the dashboard stays green.