How to Evaluate AI Agents: Why Agentic Complexity Breaks Traditional Testing
How to Evaluate AI Agents
In June 2025, Gartner predicted that over 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value, and inadequate risk controls. The same release coined a term for the noise around the category — "agent washing."
Beneath the hype is a real engineering problem: agents are fundamentally harder to evaluate than the models that power them. A chatbot produces one response you can grade. An agent reasons across many steps, calls tools, writes to systems, and makes decisions in a loop. The number of paths it can take explodes, and a single wrong turn early can cascade into a failure that looks nothing like the input that caused it.
The Error-Compounding Problem
The core reason agentic complexity breaks traditional testing is arithmetic. If an agent completes a task in n sequential steps, and each step succeeds independently with probability r, then end-to-end success is roughly r raised to the power n. Reliability that looks excellent per-step collapses over a long task.
| Task length | 90% per step | 95% per step | 99% per step |
|---|---|---|---|
| 1 step | 90% | 95% | 99% |
| 5 steps | 59% | 77% | 95% |
| 10 steps | 35% | 60% | 90% |
| 20 steps | 12% | 36% | 82% |
A model that is "95% reliable" — which sounds production-ready — completes a 20-step task barely a third of the time. This is why a strong benchmark score on single-turn tasks tells you almost nothing about agentic reliability.
Non-Determinism Makes It Worse
Even holding the task constant, the same agent can behave differently on repeat runs — and not only because of sampling temperature. A detailed technical analysis from Thinking Machines Lab, Defeating Nondeterminism in LLM Inference, traces this to a lack of batch-invariance in inference servers: identical inputs can yield different outputs depending on unrelated load. For evaluation, the implication is severe — a single benchmark run is a sample, not a measurement. You have to test repeatedly and measure consistency.
What the Benchmarks Actually Reveal
The serious agent benchmarks were built precisely to expose this.
- τ-bench (Sierra) simulates an agent using domain APIs while talking to a simulated user under real policies. It introduced the pass^k metric — the probability that all k independent attempts succeed — to measure consistency rather than luck. The headline finding: leading function-calling agents succeed on fewer than half of tasks, and consistency degrades sharply as k rises (pass^8 falls below 25% in the retail domain).
- SWE-bench Verified — a 500-task, human-validated subset released by OpenAI's Preparedness team in August 2024 (reviewed by 93 professional developers) — tests whether agents resolve real GitHub issues against the repo's own unit tests.
- GAIA measures general assistant tasks requiring reasoning, tool use, and browsing, where humans score ~92% against far lower model scores.
The pattern across all of them: average success is lower than the marketing suggests, and consistency is the real weakness.
Multi-Agent Systems Fail in Their Own Ways
Adding more agents does not solve this — it adds new failure surfaces. The UC Berkeley study Why Do Multi-Agent LLM Systems Fail? produced MAST, the first empirically grounded taxonomy of multi-agent failures: 14 distinct failure modes across three categories — specification and design issues, inter-agent misalignment, and task verification failures — derived from 200+ traces across seven popular frameworks with high inter-annotator agreement (Cohen's κ = 0.88). Many failures are not capability problems at all; they are coordination and specification problems that no amount of model intelligence fixes.
What Evaluating an Agent Actually Requires
You cannot evaluate an agent by grading its final answer alone. You have to evaluate the trajectory — every step it took to get there.
A complete evaluation checks whether each reasoning step was justified, whether tools were called correctly and with valid arguments, whether the agent stayed within its intended authority (OWASP's LLM06: Excessive Agency is the canonical risk here), and how reliably it reproduces success across many runs — not one lucky pass.
Why Human-Led Evaluation Matters Here
Automated scorers struggle with trajectories. Judging whether a sequence of decisions was reasonable — not just whether the final output looks right — is exactly the kind of contextual judgment where automated "LLM-as-judge" methods are weakest and where a qualified human evaluator is strongest. The output of that evaluation should be a structured Evidence Report: documented findings, severity, and the failure modes observed across the agent's actual behaviour.
Agents are powerful precisely because they act autonomously. That is also why the bar for evaluating them — before they act on your customers, your data, or your systems — is higher, not lower. For the infrastructure that makes agents observable enough to evaluate in the first place, see The Non-Negotiable AI Agent Stack.