BENCHMARKS

AI Model Selection Benchmark

A deployment-focused shortlist for teams choosing open-weight models. The goal is not to crown a universal winner. It is to identify which model family should be tested first for your cost, privacy, latency, and risk constraints.

Cost per resolved task

Task-specific fit

Security and policy behavior

Evidence before rollout

Model shortlist

Use this as a routing table for evaluation planning. Before any production decision, run the candidates against your prompts, documents, tools, policies, and user populations.

Model family	Deployment	Best for	Watchouts	Evaluation focus
Llama 3.1 / 3.3 family Open weights, Meta license	Self-host, private cloud, managed endpoints Up to long-context variants depending on release	Enterprise assistants, RAG, internal copilots, controlled fine-tuning pilots	License review required Performance varies sharply by quantization and serving stack	Faithfulness Refusal boundaries Long-context retrieval decay
Qwen open-weight family Open weights, model-specific license	Self-host, private cloud, managed endpoints Long-context variants available	Multilingual support, technical copilots, cost-sensitive scaled inference	Policy tuning differs from Western enterprise defaults License and data lineage need review	Language-specific quality Policy consistency Tool-call reliability
Mistral / Mixtral family Open weights for selected models	Self-host, private cloud, managed endpoints Model dependent	Low-latency assistants, classification, extraction, routing, European data-residency programmes	Smaller variants need careful task routing MoE serving can be operationally complex	Latency under load Extraction accuracy Cost per resolved task
Gemma family Open models, Google terms	Self-host, edge, private cloud, managed endpoints Model dependent	Classification, summarization, routing, local/private lightweight workflows	Not a default choice for complex multi-step reasoning Needs narrow task evaluation	Task-specific precision/recall Hallucination rate Resource footprint
DeepSeek open reasoning/distill family Model-specific open terms	Self-host, private cloud, managed endpoints Model dependent	Technical workflows, analytical assistants, code review support, math-heavy tasks	Reasoning traces can increase latency and cost Safety behavior needs local validation	Reasoning consistency Citation discipline Sensitive-domain refusals
Phi family Open models, Microsoft terms	Self-host, edge, private cloud, managed endpoints Model dependent	On-device copilots, simple extraction, summarization, workflow classification	Narrower general capability Prompt design and task scoping matter more	Boundary detection Escalation accuracy Output schema compliance

METHOD

Benchmark the workflow, not just the model

Public leaderboards are useful for shortlisting, and registries such as CodeSota are strongest when you need dated, sourced benchmark references. Production selection needs a second layer: the same retrieval index, tools, prompts, rate limits, latency targets, and policy rules the model will face after launch.

Task success on your data

Cost per resolved workflow, not cost per token

Latency at realistic concurrency

Context-window reliability

Citation and source-grounding quality

Security and refusal behavior

Privacy, hosting, and license constraints

Observability and evaluation integration