AI Incident Response: A Runbook

AI Incident Response: A Runbook

Every engineering organisation has an incident process. It was designed for outages: a service stops responding, an alert fires, an on-call engineer is paged, a fix is deployed, the graph recovers, someone writes a post-mortem. It works, because the failure is loud, discrete, and bounded in time.

AI incidents are none of those things. The system stays up. Latency is normal. Error rates are flat. The failure is that the outputs are wrong — and they have been wrong for eleven days, and forty thousand of them have already reached customers, and nobody can say which forty thousand.

That is a different animal, and it needs a different runbook.

What makes an AI incident different

It is silent. There is no exception to catch. A model that confidently produces a wrong answer emits a 200 OK.

The blast radius is retrospective. When a payments service goes down, the damage stops when the service comes back. When a model has been misclassifying since a silent provider upgrade, every decision made in that window is potentially contaminated — including the ones already acted upon. Containment does not end the incident; it only stops it growing.

The root cause is often outside your codebase. A provider changed a checkpoint. A retrieval corpus was re-indexed. An upstream data feed changed a field's encoding. A policy changed and made previously acceptable outputs unacceptable. Your git history shows nothing, because nothing you own changed.

Reproduction is probabilistic. The most maddening property. You cannot always reproduce the failing output by replaying the input, because sampling is stochastic and the system may have changed underneath you. "Works on my machine" becomes "worked that time."

The five phases

Containment is red because it is the phase teams skip. The engineering instinct is to understand the bug before touching the system. In an AI incident, every hour spent understanding is an hour of continued output, and those outputs are the harm. Contain first. Debug in the replica.

Phase 1 — Detect and triage

Signals arrive from four places, in rough order of how much they cost you: a customer complaint, an internal user, an automated monitor, or a scheduled probe replay. If most of your incidents are found by the first, your detection is your customers.

Triage answers three questions in the first thirty minutes:

• Is it real? One bad output is an anecdote. Establish a rate — sample recent outputs of the same class and count. An incident is declared on evidence, not on a screenshot in a chat channel.
• Is it bounded? Does it affect one prompt template, one document type, one language, one customer, or everything?
• Is it escalating? Is the failure rate stable or growing? A silent model swap produces a step change; a corpus drift produces a ramp.

Declare a severity, and define your severities by consequence to the affected person, not by internal disruption. A system quietly denying eligible claimants is a severity one incident even though every dashboard is green.

Phase 2 — Contain

Containment options, roughly in order of preference:

• Roll back the model or prompt to the last configuration with passing evidence. This assumes you know what that configuration was, which assumes you version it. Many teams discover during their first incident that they do not.
• Pin the provider version if the API supports it, and stop floating on a moving alias.
• Disable the affected path — a feature flag on the specific capability, not the whole product.
• Insert mandatory human review for the affected decision class. This degrades throughput and is often the only option that preserves the service.
• Fail closed. For high-consequence decisions, no output is better than a wrong one. This should be a pre-agreed option, not a debate held at 2am.

Every one of these must be exercisable in minutes and must have been tested when there was no incident. A rollback path you have never used is not a rollback path.

Phase 3 — Reconstruct

This is the phase that separates organisations that can answer a regulator from organisations that cannot.

You must bound the blast radius: which outputs, produced when, for whom, were affected — and which of them were acted upon. Answering that requires that you logged, at the time of each decision: the input, the output, the model and version, the prompt version, the retrieval context, the rubric or policy in force, and whether a human reviewed it.

If you did not log those, you cannot reconstruct, and your incident report will contain the sentence every counsel dreads: we are unable to determine which customers were affected. The time to fix that is not during the incident.

Reconstruction produces three artefacts: a window (when did this start — usually earlier than the first complaint), a population (which decisions fall inside it), and a contamination estimate (of that population, what proportion is actually wrong, established by sampling and human review rather than by assumption).

Phase 4 — Remediate

Remediation has two halves, and organisations reliably do only the first.

Fix the system. Correct the prompt, restore the model, repair the corpus, add the guardrail. Address the root cause, not the symptom — if a provider silently upgraded, the root cause is that you were floating on an unpinned alias, not that the new checkpoint is worse.

Fix the harm already done. Re-run the contaminated population. Reverse the decisions that were wrong. Notify the people affected. This is the expensive half, it is the half that has legal and regulatory obligations attached to it, and it is the half that gets deferred into a backlog ticket that is never picked up.

An incident is not closed when the system is correct. It is closed when the people harmed by the incorrect period have been made whole.

Phase 5 — Re-test before you restore

Do not restore service on the strength of "it looks fine now." The exit criterion is evidence: the affected scenarios re-evaluated, against the rubric, by people who did not write the fix.

This is the point at which independent evaluation is worth what it costs. A team that has spent three days fixing an incident is the least well-positioned group in the company to judge whether it is fixed — they are exhausted, they are invested in the fix, and they have been staring at the failing cases long enough to have stopped seeing them. An independent evaluation under a defined rubric, executed by credentialed Evaluators, with gold tasks confirming the evaluators themselves are calibrated, produces a verdict that is defensible to a regulator precisely because it was not produced by the people who caused the problem.

The output is a versioned Evidence Report: what was evaluated, against what standard, with what result, with severities and root causes attached. That is the document you will be asked for in six months, and it is much easier to have produced it at the time than to reconstruct it later.

Who does what

The unfamiliar row is the domain expert, and it is the one most often missing from the bridge. In a conventional outage, engineers can see the failure. In an AI incident, engineers frequently cannot tell whether an output is wrong, because judging it requires clinical, legal, or financial knowledge they do not have. Put that person on the call in the first hour, or spend the first day arguing about whether there is an incident at all.

What to do tomorrow

Run a tabletop. Pick a real scenario — "our provider silently upgraded the model and our extraction quality dropped 8% three weeks ago" — and walk it. You will discover, in about twenty minutes, whether you can name the last known-good configuration, whether you can enumerate the affected decisions, and whether anyone knows who declares the incident.

Then fix the two things the tabletop will certainly expose: the logging you need for reconstruction, and the rollback path nobody has ever pulled.

The incident is going to happen. The only variable you control is whether, on the day, you can say which customers were affected.