Evaluating AI Coding Assistants

Evaluating AI Coding Assistants

Coding assistants are the most widely deployed and least rigorously evaluated category of enterprise AI. The evaluation, where it exists at all, tends to consist of a benchmark number quoted from a vendor page and a developer survey asking whether people feel faster.

Both of those measure something. Neither measures the thing that matters, which is the quality of what actually lands in your repository and the total cost of getting it there.

Plausibility is not correctness, and the gap is the whole problem

A traditional buggy program looks buggy. Variables are misnamed, structure is odd, the error is visible in the shape of the code. An experienced reviewer's eye is trained on exactly those signals, and the training works because a human who does not understand the problem usually writes code that looks like a human who does not understand the problem.

Generated code inverts this. It is idiomatic by construction. It has sensible names, consistent style, plausible error handling, and a docstring. It looks like code written by someone who understood the requirement. When it is wrong, it is wrong underneath an entirely convincing surface — an off-by-one in a boundary the tests do not cover, a currency handled as a float, a timezone assumed, a rare branch that silently swallows an exception.

The reviewer's heuristics, honed over a career, are calibrated to a correlation between surface quality and underlying correctness that generated code has broken. This is the central fact about evaluating coding assistants, and it means the reviewer is systematically over-trusting exactly the output that most warrants suspicion.

Notice that "tests pass" and "looks right" both route to the same shallow review, and that the three red boxes are all downstream of an approval that felt entirely reasonable at the time.

What the benchmarks do not measure

Public coding benchmarks score a model's ability to produce a function that passes hidden unit tests, given a self-contained problem statement. That is a real capability and it is nearly orthogonal to enterprise software work.

Real changes are not self-contained. They span files. They must conform to conventions the benchmark has never seen, use the internal library rather than reinventing it, respect an authorization pattern established three years ago, and preserve behaviour that no test protects. The benchmark problem is a puzzle with a known answer; the enterprise problem is a change to a system nobody fully understands, in which the hardest part is knowing what not to touch.

There is also a subtler distortion. Benchmark problems are, by design, the kind of problem that can be specified in a paragraph and verified by assertion. Assistants are strongest exactly there. The tasks where they are weakest — long-range consistency, architectural judgement, knowing that a requirement is ambiguous and asking — are the tasks no benchmark contains, because they cannot be auto-graded.

So a high benchmark score tells you the assistant is good at the easiest quartile of your work. It says nothing about the rest.

Four dimensions worth actually scoring

A defensible evaluation of a coding assistant is run on your codebase, against your tickets, and scores at least four independent dimensions. Collapsing them into one number destroys the information.

Functional correctness against intent. Not "do the tests pass" — the assistant may have written the tests, and a test written by the same process that wrote the bug encodes the bug. The question is whether the change does what the ticket asked, including the parts the ticket implied. This requires a qualified engineer reading the diff against the requirement, which is why it is expensive and why it is skipped.

Security posture of generated code. Assess against a concrete list: injection through string-built queries, missing authorization on a new endpoint, secrets in code or logs, unsafe deserialisation, weak or hand-rolled crypto, path traversal, missing validation on external input, dependencies that are outdated, abandoned, or hallucinated outright. The last one deserves its own attention — an assistant confidently importing a package that does not exist is a supply-chain vulnerability waiting for someone to register the name.

Codebase fit. Does the change use the conventions and the internal abstractions that already exist, or does it invent a parallel implementation because it could not see the rest of the repository? Duplication generated in seconds is maintained for years, and it is the cost that never appears in a productivity metric.

Review burden. How long did a competent engineer take to review it, and how many round trips before merge? This is the dimension that determines whether the assistant produced value at all.

The metric that decides the business case

Time-to-first-draft is the number vendors report and the number that flatters everyone. It is also, in most organisations, close to irrelevant, because writing the first draft was never the bottleneck.

The number that decides the business case is net cycle time to merged, correct code, including review, rework, and the defects that escape. An assistant that produces a plausible 300-line diff in twenty seconds and then consumes ninety minutes of senior review has not accelerated anything. It has moved effort from a person who enjoys writing code to a person who does not enjoy reading it, and it has moved it from the cheaper activity to the more expensive one.

Worse, review effort does not scale linearly with diff size — it scales worse, because a large diff exceeds the reviewer's working memory and the reviewer starts sampling. Sampling a diff means approving code nobody read. The assistant's greatest strength, volume, is therefore also the mechanism by which it defeats the control that was supposed to make it safe.

This suggests a deployment rule that is unpopular and correct: cap the size of a generated change. Not because large changes are impossible, but because a change too large to review is a change that will not be reviewed.

Measuring what escapes

The dimensions above measure code at the moment of review. The honest evaluation also measures what got past it.

Track the provenance of merged changes and then look at what happens to them downstream: defect rate per changed line, incident attribution, rate of subsequent revert or hotfix, and — most revealingly — the rate at which a change is modified again within thirty days. Code that keeps needing repair was not finished; it was accepted.

Sample merged assistant-generated changes at random for a deeper independent review, in the same way you would sample any high-volume process whose failures are silent. The finding that a random sample of forty merged diffs contained three authorization gaps is a fact you can act on. The absence of complaints is not.

What independent evaluation adds

Teams evaluate their own coding assistants with a structural handicap: the people best placed to judge the output are the people whose productivity the tool is meant to improve, and who have often already told their leadership that it does. That is not dishonesty. It is an incentive.

Independent evaluation places the same diffs in front of credentialed Evaluators scoring against an explicit rubric, with anchored levels for correctness, security, and codebase fit. Where two evaluators disagree on a diff, the disagreement is measured and adjudicated rather than averaged away — and a persistent disagreement usually means the rubric dimension is underspecified, which is itself worth knowing. Gold tasks with known injected defects give you a detection rate: proof, rather than assertion, that your review process catches the class of failure you are worried about. The result is a versioned Evidence Report that says what the assistant does to your codebase, with severities and root causes, rather than what it scored on someone else's puzzle set.

What to do tomorrow

Pull the last fifty merged changes that an assistant substantially wrote. Have a senior engineer who did not review them the first time read a random ten against the original tickets, scoring correctness, security, and fit separately. Then measure the review time those diffs actually consumed and compare it with the review time for comparable human-authored changes.

You will end up with three numbers: what the assistant gets wrong, what your review catches, and what it costs. Those are the only three numbers the decision needs, and none of them are on a benchmark page.