How to Manage AI Costs in Production
How to Manage AI Costs in Production
Most AI cost programmes start in the wrong place. Teams look at token prices, shorten prompts, and ask engineers to cache more aggressively. Those tactics help, but they miss the unit that finance and product leaders actually care about: cost per resolved workflow.
An AI support assistant that costs less per token but escalates twice as many conversations is not cheaper. A model that generates a low-cost draft but requires a senior reviewer to rewrite it is not cheaper. A RAG system that retrieves the wrong context and forces users to retry is not cheaper. The bill you need to manage is the complete cost of getting the job done.
The cost stack
Production AI cost has seven layers:
| Cost layer | What to measure | Common failure |
|---|---|---|
| Inference | Input tokens, output tokens, latency | Optimising token price while quality drops |
| Retrieval | Embedding, vector search, reranking | Sending too much context to the model |
| Tool calls | API usage, database reads, external services | Letting agents loop or retry without budget limits |
| Human review | Review minutes, escalation rate, rewrite rate | Counting model cost but ignoring labour cost |
| Failed attempts | Retries, abandons, unresolved sessions | Treating every request as a success |
| Evaluation | Sampling, judge models, expert scoring | Not measuring whether cheaper outputs still pass |
| Governance | Logging, retention, audits | Rebuilding evidence later at higher cost |
The cheapest system is the one that resolves the workflow correctly with the least total effort.
Route tasks before you route models
A model router should not start with model names. It should start with task classes:
- Classify and route — small model or rules first.
- Extract structured fields — small or mid-size model with schema validation.
- Summarise low-risk content — mid-size model with length and citation checks.
- Answer from enterprise documents — RAG model with retrieval quality checks.
- Reason across policy, finance, medical, or legal content — larger model plus human review threshold.
- Act with tools or write to systems — agent harness with budget limits, approvals, and trace-level evaluation.
This lets you pay for expensive reasoning only when the task actually requires it.
Use budget gates inside the agent
Agents need explicit budgets. A production agent should know:
- maximum tool calls per task
- maximum retrieval chunks
- maximum model retries
- maximum latency
- maximum spend per session
- escalation threshold when confidence or grounding drops
Without these gates, cost growth looks like "usage growth" until someone inspects traces and finds tool loops, over-retrieval, or repeated failed attempts.
Measure cost alongside quality
Every candidate model should be evaluated on a paired scorecard:
| Metric | Why it matters |
|---|---|
| Cost per resolved task | Captures retries, escalation, and failed attempts |
| Pass rate against task rubric | Prevents cheap but unusable outputs |
| Human edit distance | Shows whether reviewers are really saving time |
| Citation accuracy | Reduces downstream correction cost |
| Latency at concurrency | Catches infrastructure cost before launch |
| Escalation precision | Avoids sending easy tasks to humans |
Cost optimisation without quality measurement creates hidden risk. Quality optimisation without cost measurement creates prototypes nobody can afford to scale.
The operating cadence
Run cost reviews weekly during launch and monthly after stabilisation. Each review should answer:
- Which task classes consumed the most spend?
- Which model produced the highest cost per resolved workflow?
- Which prompts caused retries or long outputs?
- Which retrieval queries pulled unnecessary context?
- Which failures should be routed to a smaller model, larger model, or human?
The goal is not to minimise AI spend. The goal is to buy the right amount of intelligence for each workflow.
Where evaluation fits
You cannot safely reduce cost unless you know what quality you are allowed to lose. That is why cost management belongs inside the evaluation programme. A cheaper model is acceptable only if it still passes the task rubric, safety checks, citation checks, and escalation policy.
Use the AI Model Selection Benchmark to shortlist candidates, then run a production-style evaluation on your data before changing the routing policy. For high-stakes systems, capture the decision in an Evidence Report so finance, engineering, risk, and compliance are looking at the same proof.