How Many AI Decisions Must You Actually Review?
How Many AI Decisions Must You Actually Review?
Every team deploying AI at volume arrives at the same question, usually in a budget meeting. The system makes fifty thousand decisions a month. Human review costs real money. How many do we have to look at?
The answers offered are almost always wrong in one of two directions. "Review everything" is not an answer; it is a refusal to deploy, because if humans check every output the AI has saved nobody any work. "Review a hundred a month, the ones the system flagged" is worse, because it feels like diligence while systematically hiding the failures that matter.
There is a defensible middle. It requires accepting three ideas: coverage is a statistical claim, not a percentage; risk should determine sampling rate, not volume; and a sample drawn from flagged outputs tells you nothing about unflagged ones.
The bias at the centre of the problem
Start with the failure that most review programmes are built on.
A system produces outputs. A confidence score, a guardrail, or an escalation rule flags a subset as suspicious. Reviewers review the flagged subset. Reviewers find errors — of course they do, the flag selected for them. Management sees a review programme finding errors and concludes it is working.
What has actually happened is that you have measured precision on the flagged set and learned nothing whatsoever about recall. You know how often the system's suspicion was justified. You have no estimate at all of how often it was confidently, quietly wrong — which is the only failure mode that reaches a customer, because the flagged ones were caught.
The dashed line is the entire argument. Without a random sample of the unflagged population, the red boxes stay red forever. You cannot compute a false-negative rate from a sample that excludes negatives.
So the first rule of sizing a review programme: some portion of the sample must be drawn at random from everything, including the outputs the system is proud of.
What a sample actually buys you
Sampling is not about proportions. The instinct to review "1% of volume" is arithmetic, not statistics, and it produces absurd results: a system doing five million decisions gets fifty thousand reviews it does not need, while one doing two thousand gets twenty, which cannot detect anything.
What determines the precision of an estimate is the absolute size of the sample, not its ratio to the population — for any population large relative to the sample, which describes essentially every AI deployment. A random sample of a few hundred outputs gives you a similar confidence interval on the error rate whether the population is ten thousand or ten million.
This is liberating and counter-intuitive, and it means the sizing question is not "what fraction?" but three sharper questions:
What are you trying to detect? Detecting a common failure (say, one in ten outputs) needs far fewer samples than detecting a rare-but-catastrophic one (one in a thousand). If a failure mode is rare, a small random sample will usually return zero instances — and zero instances is not evidence of safety, it is evidence of an underpowered sample. Be explicit about the smallest failure rate you need to be able to see.
How precisely? "The error rate is somewhere between 2% and 12%" may be enough to decide whether to keep a pilot running. It is not enough to certify a high-risk system. Tighter intervals cost more samples, and the cost grows steeply: halving the width of the interval roughly quadruples the sample.
With what consequence? The purpose of the number is to trigger a decision. If a 3% error rate and a 6% error rate would lead you to do exactly the same thing, do not pay for the sample that distinguishes them.
Work backwards from the decision. If you cannot name the action that a given result would change, you are buying reassurance, not information.
Stratify by risk, not by volume
Uniform random sampling across all traffic is the right foundation and the wrong finished product, because it allocates your most expensive reviewers in proportion to how common a decision type is rather than how much it matters. A high-volume, low-consequence path (routing a support ticket) absorbs the majority of the sample; the low-volume, high-consequence path (denying a claim, escalating a clinical alert) gets a handful of reviews.
Stratified sampling fixes this. Partition the decision population into strata by consequence, then set an independent sampling rate per stratum:
| Stratum | Consequence of an undetected error | Sampling approach |
|---|---|---|
| Irreversible or safety-critical | Harm to a person; unrecoverable action | Census or near-census — review every one |
| Adverse to an individual | Denial, downgrade, adverse determination | High rate; sufficient sample per period to bound the error rate |
| Reversible and material | Wrong output that a human will likely catch downstream | Moderate random rate |
| Low-stakes, high-volume | Cosmetic, easily corrected | Low random rate, sufficient to detect drift |
Two properties make this work. Each stratum is sampled at random within itself, so each yields an unbiased estimate of its own error rate. And the strata are defined by consequence, so your reviewer hours concentrate where an error is expensive rather than where outputs are plentiful.
Note that the top stratum is not sampled at all. If the consequence of an undetected error is irreversible harm, the correct sampling rate is one hundred percent, and if that is unaffordable then the honest conclusion is that the system is not ready to make that decision without a human in the loop.
The three-part sample
A mature review programme is not one sample. It is three, running concurrently, each answering a different question.
The random stratum — an unbiased sample across all outputs, stratified by consequence. This is the only component that can estimate your true error rate and the recall of your automated flags. It is non-negotiable and it is the part budget pressure always attacks first, because it spends reviewer time on outputs that mostly turn out to be fine. That is the point: the finding that 96% of unflagged outputs are correct is a real, defensible, reportable result. It is what lets you say a number out loud.
The targeted stratum — the flagged, low-confidence, guardrail-tripping, user-complained-about outputs. This is where you learn what breaks and why. It is rich in findings and worthless for rate estimation. Never let a metric derived from this stratum be reported as a system-wide error rate.
The adversarial stratum — deliberately constructed hard cases, edge inputs, and known-difficult scenarios. This finds failure modes that neither random nor flagged sampling will surface, because production traffic simply does not contain them yet.
Report the three separately. The most common reporting failure in AI assurance is a single blended number computed across a sample that mixed random and flagged outputs, which is neither an error rate nor a finding density but a meaningless average of the two.
Consistency: the sample is only as good as the reviewer
A sample sized correctly and reviewed inconsistently is a false comfort. If two qualified people reviewing the same output reach different verdicts, your measured error rate is partly a measurement of your reviewers.
This is where the mechanics of structured evaluation earn their cost. A rubric with anchored scale points constrains interpretation. Gold tasks — outputs with an agreed correct assessment, injected into the review queue — give you a per-reviewer calibration score. Overlapping assignments and an inter-rater agreement statistic tell you whether the rubric itself is ambiguous. Where reviewers genuinely disagree, an adjudication step resolves the case and, more importantly, exposes the rubric dimension that needs rewriting.
Without these, you do not have a review programme. You have a queue.
What to do tomorrow
Write down your decision strata by consequence. Set a census rule for the top one. For each remaining stratum, name the smallest failure rate you must be able to detect and the decision that a result would change, then size the random sample to that — not to a percentage of volume. Run the targeted and adversarial samples alongside it, and never merge their numbers.
Then take the result and put a confidence interval on it. A review programme that produces a point estimate with no interval is reporting a feeling. One that produces an interval, from a random sample, drawn from a defined population, reviewed against a versioned rubric by calibrated evaluators, is producing evidence.