What Is an Agent Harness? The Hidden Variable in AI Agent Performance
What Is an Agent Harness?
Two teams run the same model on the same benchmark and report scores 20 points apart. Neither is lying. The difference is the harness — the layer of code wrapped around the model that turns a text generator into something that can act. It is the single most underappreciated variable in AI agent performance, and it is why most published agent numbers are close to meaningless on their own.
Defining the Harness
A raw language model does one thing: it predicts text. To make it an agent, you wrap it in scaffolding that lets it perceive, decide, and act in a loop. That scaffolding is the harness.
Change any of these — how tool calls are parsed, how many times the agent retries, how context is trimmed when it overflows, how the loop decides it is done — and you change the agent's behaviour, often dramatically, with the model held completely constant.
The Evidence: Same Model, Different Score
The clearest proof comes from SWE-bench Verified, which deliberately separates the model from the scaffold it runs in. In OpenAI's own reporting, GPT-4o scored 33.2% on the Verified set — but the broader SWE-bench results showed that the open-source "Agentless" scaffold roughly doubled the previous best score on the benchmark. Same class of model; a different harness; a completely different outcome.
The lesson for anyone buying, building, or comparing agents is blunt: a benchmark number without its harness is not a result. When a vendor cites an agent's score, the questions that matter are which harness produced it and whether your harness resembles it at all. Comparisons that don't hold the scaffold constant are comparing two different systems.
Where Harnesses Break
Because the harness is code, it has its own failure modes — independent of the model's intelligence:
| Harness component | How it fails |
|---|---|
| Tool-call parsing | Malformed or hallucinated calls accepted without validation |
| Control loop | Infinite retries, premature stops, no error recovery |
| Context management | Critical history silently truncated as the window fills |
| Orchestration | Sub-agents misaligned, talking past each other |
| Permissions | More tools/authority than the task needs — OWASP Excessive Agency |
A connected risk lives at the harness's edge: tools themselves. The Model Context Protocol (MCP), introduced by Anthropic in November 2024, standardised how agents connect to external tools — but it also created a new attack surface. Tool poisoning and indirect prompt injection through tool descriptions let malicious instructions enter the model's context as trusted input. The harness, not the model, is where these are caught or missed.
The Simplest Harness That Works
More scaffolding is not better scaffolding. Anthropic's Building Effective Agents (December 2024) draws a useful line between workflows — LLMs orchestrated through predefined code paths — and agents, where the model dynamically directs its own process. Its core advice is to find the simplest design that works and add agentic complexity only when it demonstrably improves outcomes. Every layer of harness you add is another layer that can fail, and another variable your evaluation has to account for.
Why This Matters for Evaluation
If the harness determines performance, then evaluating the model alone is evaluating the wrong thing. A credible AI agent evaluation tests the deployed system — model and harness together, under conditions resembling production: real tool integrations, realistic context loads, adversarial inputs against the tool layer, and repeated runs to measure consistency rather than a single pass.
That is the difference between a benchmark score and an Evidence Report. One tells you how a model did in someone else's harness on someone else's day. The other tells you how your agent behaves in your harness — which is the only thing that matters once it is acting on your customers and your data. For a deeper look at why agentic complexity demands this, see How to Evaluate AI Agents.