Evaluating Voice AI: Latency, Recognition Error, and Escalation
Evaluating Voice AI: Latency, Recognition Error, and Escalation
Voice agents are being deployed into the highest-consequence customer channel most organisations have, and they are routinely evaluated by reading transcripts.
That is a category error, and it is the reason so many voice deployments test well and perform badly. The transcript is not the call. It is a lossy reconstruction of the call, produced by the very component most likely to have failed, with the timing removed — and timing is half of what makes a conversation work.
A voice system is a pipeline, and every stage has its own failure mode, its own error budget, and its own way of destroying the stages downstream of it.
The pipeline is the system
Evaluating only the LLM box means evaluating one of five stages, and not the one that usually breaks. Worse, it means evaluating that box on clean input — the transcript as it should have been — when in production it receives the transcript as it was.
Recognition error is not uniform, and that is the whole problem
Word error rate is the standard metric and it is close to useless on its own, because it treats every word as equally important. A speech recogniser that drops a "the" and one that mishears an account number can post identical WER.
What matters is error on the tokens that carry the decision: names, addresses, account and policy numbers, dates, amounts, medication names, yes and no. A single-digit error in an amount is a different event from a filler word lost in noise, and any metric that averages them together is hiding the failures that produce complaints.
Two further asymmetries deserve explicit measurement, because they turn a quality problem into a fairness problem:
Recognition quality is not equally distributed across speakers. Accent, dialect, age, speech impairment, and non-native speech all affect recognition, and they affect it in ways that correlate with protected characteristics. A voice system whose recognition error is materially worse for one group is a system that delivers worse service to that group — longer calls, more repetition, more failed authentications, more escalations. This is an equity finding, not just an accuracy one, and it will not appear in an aggregate WER because the affected population is, by definition, a minority of the sample.
Recognition quality is not equally distributed across conditions. Mobile in a car, a call from a hospital ward, a bad line, a background television. Evaluate on studio-quality audio and you have measured the best day of the system's life.
Then there is the failure that sits underneath all of it: confidence is usually discarded at the boundary. The recogniser knows it was unsure. The language model receives a clean string of text with no indication that the account number is a guess, and proceeds with total assurance. A voice stack that does not propagate recognition confidence into the downstream turn has thrown away the one signal that could have triggered a confirmation.
Latency is a correctness property
In text, a slow response is an annoyance. In speech, it is a semantic event.
Human conversation runs on turn-taking gaps of a couple of hundred milliseconds. Exceed that meaningfully and the caller does not perceive a slow system — they perceive that they were not heard, and they repeat themselves. That repetition arrives mid-generation, and now the system is processing a duplicate utterance while speaking a response to the first one. The conversation degrades from a single delay, and the delay is not the failure; the caller's entirely reasonable reaction to it is.
So latency must be measured end to end — the wall-clock gap between the caller finishing and hearing the first syllable back — and measured as a distribution, not a mean. The p95 is where the calls go wrong. The mean is where the demo went right.
The related and under-appreciated control is endpointing: the decision about when the caller has finished speaking. Set it aggressively and the system interrupts people mid-sentence, which is infuriating and disproportionately affects slower speakers, people who pause to think, and anyone reading a number off a card. Set it conservatively and every turn carries dead air. There is no setting that is correct for all callers, which means endpointing behaviour must be evaluated as a policy with a fairness dimension, not tuned as a constant.
Escalation is the feature, not the fallback
The most consequential design surface in a voice agent is the path to a human, and it is treated as an afterthought in almost every deployment.
Get it right and the system's failures are recoverable: the caller is annoyed for thirty seconds and then handled. Get it wrong and a routine failure becomes a trapped, angry customer repeating "agent" into a machine that keeps offering to help — which is, reliably, the interaction that ends up on social media.
Four properties define a working escalation path.
A trigger that fires on the right signals. Repeated low recognition confidence. The caller repeating themselves. Explicit request for a human, in any phrasing — and the phrasings people actually use are rude, exasperated, and not in your intent list. Detected distress. Any topic on the out-of-scope list: complaints, vulnerability, safeguarding, bereavement, anything with a legal or clinical consequence.
A hard stop rule. After N failed attempts at the same step, the system escalates. Not tries again. This is the single most valuable line of configuration in a voice deployment, and its absence is the source of most loop-of-doom incidents.
Context that survives the handoff. The human must receive who the caller is, what was attempted, what was recognised and with what confidence, and what the agent already told them. An escalation that makes the caller start again has converted one failure into two, and the second one is the one they will remember.
An unconditional escape hatch. Something the caller can always say to reach a person, which is not gated on the agent understanding why. If the agent must comprehend the request in order to honour it, then the callers who most need it — the ones the agent is failing to understand — are precisely the ones who cannot use it.
Evaluating what the transcript cannot show
A defensible voice evaluation is scored on audio, by people, with the timing intact.
That means credentialed Evaluators listening to real or realistic calls and scoring against a rubric with dimensions the transcript cannot capture: was the caller interrupted, was there dead air, did the system confirm critical entities before acting, did it recover gracefully when it misheard, did it escalate when it should have, was the synthesised speech intelligible when reading a reference number aloud. It means stratifying the evaluation set by accent, by line quality, by background noise, and reporting per-stratum results rather than an average that conceals them. It means scenario sets that include the difficult calls — the confused caller, the angry one, the one who hands the phone to someone else mid-call, the one with a child shouting in the background.
Because two evaluators can hear the same call differently, agreement between them is measured and disagreements are adjudicated rather than averaged. And the result — findings, severities, root causes, per-stratum performance — belongs in a versioned Evidence Report, because a voice system that works well for the median caller and fails for an identifiable group is a specific, reportable, remediable finding, not a rounding error.
What to do tomorrow
Pull twenty recorded calls that ended in an escalation or an abandon, and listen to them. Not the transcripts — the audio. Count how many failed because the model said something wrong, and how many failed because the system misheard, interrupted, stalled, or refused to let the caller out.
The ratio will surprise you, and it will tell you which part of your stack you have been evaluating and which part has been failing your customers.