How to Select the Right AI Model
How to Select the Right AI Model
Model selection is often treated like a leaderboard exercise. Pick the highest score, plug it into the product, and optimise later. That works for demos. It fails in production because your users do not interact with a benchmark. They interact with a full system: prompts, retrieval, tools, guardrails, latency budgets, review queues, and business rules.
The right model is the cheapest model that passes the complete deployment test.
Start with constraints, not model names
Before comparing models, write down the constraints:
| Constraint | Question to answer |
|---|---|
| Task | What exactly must the model do? |
| Risk | What harm occurs if it is wrong? |
| Data | Can inputs leave your environment? |
| Latency | How fast must the response be? |
| Context | How much source material must it handle? |
| Governance | What evidence will an auditor or buyer ask for? |
| Cost | What is the acceptable cost per resolved workflow? |
Only after this should you shortlist models.
Use a three-tier model strategy
Most production systems should not use one model for everything.
Tier 1: Small and cheap. Use for classification, routing, extraction, summarisation, and low-risk transformations.
Tier 2: Mid-size and reliable. Use for standard customer-facing answers, RAG responses, structured drafting, and common internal copilots.
Tier 3: Expensive and capable. Use for high-risk reasoning, ambiguous cases, tool-heavy agents, and tasks that require better judgment or escalation.
Routing between tiers usually creates more savings than switching providers.
Evaluate open-weight models differently
Open-weight models are attractive because they can reduce vendor lock-in, support private deployment, and give teams more control over fine-tuning. But they also move more responsibility onto you.
For each candidate, evaluate:
- license fit for your commercial use
- hosting and security boundary
- latency on your serving stack
- quantization impact
- refusal and safety behavior
- multilingual performance if relevant
- quality after your system prompt, retrieval, and tools are attached
Never assume the benchmark score survives your harness. Quantization, prompt templates, context length, retrieval quality, and tool schemas can all change performance.
Build the model trial
A useful model trial has five parts:
- Golden task set — real examples with expert-written expected outcomes.
- Adversarial set — prompt injection, ambiguity, missing data, conflicting sources.
- Cost and latency run — realistic concurrency, not a single local test.
- Human review — domain experts score outputs and measure edit effort.
- Routing policy — decide when to use small, mid-size, large, or human review.
The output should be a decision memo, not a vibe check.
The model selection scorecard
| Dimension | Pass condition |
|---|---|
| Accuracy | Meets task rubric on representative data |
| Grounding | Claims are supported by source material |
| Safety | Refuses or escalates harmful requests correctly |
| Security | Resists injection and tool misuse |
| Privacy | Meets data residency and retention rules |
| Latency | Performs under realistic load |
| Cost | Meets cost per resolved workflow target |
| Operability | Can be traced, monitored, and evaluated continuously |
If a model fails any mandatory dimension, a higher public benchmark score does not matter.
The practical shortlist
Use public benchmarks to reduce the search space. Then use your own evaluation to choose. We built the AI Model Selection Benchmark as a practical starting point for open-weight model families and the evaluation tests each one should face.
The decision is not "which model is best?" The decision is "which model is good enough, controllable enough, cheap enough, and proven enough for this workflow?" That is the difference between model shopping and production AI governance.