Evaluating Recommender and Ranking Systems

Evaluating Recommender and Ranking Systems

Ask an organisation to list its AI systems and you will get the chatbot, the copilot, the document extractor, and — if the list is honest — the fraud model. You will very rarely get the ranker.

Yet the ranker is often the most consequential model in the building. It decides which job adverts are shown, which candidates a recruiter sees on page one, which products are surfaced, which content a user is fed, which cases an investigator reviews first. It is a decade older than the language model, it touches vastly more people, and it is governed by a team whose success metric is engagement.

It is also the model with the most treacherous evaluation problem in all of machine learning, for one reason: its training data is a record of its own past behaviour.

The feedback loop

A recommender shows ten items out of ten thousand. A user clicks one. The system learns that the clicked item was good. It shows it more. More users see it, some click, and it is reinforced.

Now consider the item ranked 4,000th. Nobody clicked it — because nobody saw it. To the training pipeline, "not shown" and "shown and rejected" produce the same signal: no click. The model concludes the item is bad, and demotes it further.

The loop closes on itself, and the offline evaluation closes it tighter. Because offline evaluation replays logged interactions — the interactions the current model produced — a new candidate model is scored on how well it agrees with the incumbent. A genuinely better ranker that surfaces items the incumbent never showed will score worse offline, because the ground truth contains no positive labels for items nobody was given the chance to click.

This is the single most important thing to understand about recommender evaluation: your offline metric has a structural bias toward the model you already have. It is not a neutral referee. It is the incumbent's home crowd.

What offline metrics can and cannot tell you

The standard toolkit — precision@k, recall@k, NDCG, MAP, MRR, hit rate — measures agreement with logged behaviour, ranked. It is genuinely useful for one thing: catching a candidate model that is catastrophically worse. If NDCG collapses, do not ship. Offline metrics are a cheap, fast filter, and they should be run on every change.

They cannot tell you:

• Whether an item nobody saw would have been clicked (the missing-labels problem above).
• Whether a click was a good click. Clicks are correlated with quality and also with outrage, clickbait, and the top-left corner of the screen. Position bias alone accounts for a substantial share of click probability, which means your relevance model is partly a model of where things appear.
• Whether the recommendation was appropriate — a technically relevant recommendation can be inappropriate (gambling content to someone recovering, weight-loss products to someone with a disclosed eating disorder, high-interest credit to someone in arrears). No ranking metric has a term for this.
• Whether the system is fair in exposure. Two providers, two sellers, two candidates with equal merit may receive wildly unequal exposure, and every relevance metric in the list above is indifferent to that.

Counterfactual and off-policy estimators — inverse propensity scoring and its variants — partially correct for the exposure bias, and they are worth the effort. But they need logged propensities, they have high variance when the new policy diverges from the old, and they cannot invent evidence about items with zero exposure. They narrow the gap. They do not close it.

Online, and its own dishonesty

So you run an A/B test. Online evaluation is the closest thing to ground truth, because the new model actually exposes items and users actually respond.

Three failures recur.

The metric is a proxy that you have begun to optimise past the point of usefulness. Engagement rises. Session length rises. And what you have shipped is a system that is better at capturing attention, which is not the same thing as better at serving the user — and in some domains is the direct opposite. Any organisation that has run this experiment long enough knows that the metric that goes up is not always the one you wanted.

The horizon is too short. A two-week test measures the novelty response. The effects that matter — narrowing of the content a user is shown, erosion of trust, the slow disappearance of the long tail, the supply-side exit of providers who stopped getting exposure — take months and are invisible in the experiment window. Some of the most damaging recommender harms are only visible over horizons longer than any A/B test anyone is willing to run.

Interference. Recommender A/B tests violate the independence assumption. Treatment users change the inventory, the popularity signals, and the seller behaviour that control users experience. The measured lift is frequently an artefact of the two arms competing over shared state.

The missing evaluation: quality, not agreement

Between the offline metric that measures agreement with the past and the online metric that measures short-horizon engagement, there is a question neither one asks:

Was this a good recommendation?

Not "did the user click it." Not "did the incumbent also rank it highly." Was it, in the judgement of a qualified person who understands the domain and the user's context, an appropriate, useful, non-harmful thing to put in front of that person?

That is a human judgement, and it is exactly the kind of judgement structured evaluation exists to make rigorous. Sample the ranked lists actually served. Score them against a rubric with anchored dimensions — relevance to the stated need, appropriateness given known user context, diversity of the slate, presence of harmful or exploitative items, exposure concentration. Have credentialed Evaluators with domain background do the scoring. Inject gold tasks to confirm the evaluators are calibrated. Measure inter-rater agreement, adjudicate the disagreements, and record the result in a versioned Evidence Report.

This is not a replacement for offline or online metrics. It is the third leg, and it is the only one that can detect a system that is efficiently, measurably, and profitably recommending the wrong things.

Classical ML is in scope, and is usually out of scope

The governance point. Most AI assurance programmes stood up since 2023 were scoped around generative AI, because that is what alarmed the board. The credit model, the churn model, the fraud scorer, the ranker, the pricing engine — these predate the programme, they are owned by teams who have run them for years, and they are frequently not in the register at all.

That is exactly backwards relative to regulatory exposure. A ranking system that determines which candidates a recruiter sees is making a consequential decision about employment, and it does not become less consequential for having been built with gradient-boosted trees instead of a transformer. Regulators define scope by consequence, not by architecture. A model that decides who sees a job advert sits squarely in the high-risk category under the EU AI Act's employment provisions whether or not anyone on the team thinks of it as "AI."

The uncomfortable question for most governance leads: when was the ranker last evaluated by anyone who did not own its engagement metric?

What to do tomorrow

Add every ranking and recommendation system to the AI register, scoped by consequence rather than by architecture. Then, for the most consequential one, do the thing its offline metrics cannot: take a sample of the slates it actually served last week, and have people qualified in the domain score them against a rubric for relevance, appropriateness, and exposure fairness.

Measure exposure concentration while you are there — what share of impressions goes to the top 1% of items, and what has happened to that number over the last year. A ranker eating its own tail shows up in that one figure long before it shows up in NDCG.

Your offline metric will never tell you the model is wrong. It was trained by the model.