BENCHMARKS

Industry AI Benchmarks

Generic model rankings do not answer industry deployment questions. A healthcare summarizer, a credit-policy copilot, and a legal research assistant need different tasks, failure modes, reviewers, and pass thresholds.

Candidate model families

These are candidate families to route into sector-specific evaluations. Published scores should only appear after a dated, source-controlled run on disclosed task sets.

GPT family

Claude family

Gemini family

Llama open-weight family

Qwen open-weight family

Mistral / Mixtral family

DeepSeek family

Gemma / Phi small-model families

Industry	Benchmark task set	Primary metrics	Failure modes	Reviewer profile
Healthcare High-risk, clinical safety sensitive	Clinical summarization Triage support Drug interaction explanation	Clinical correctness Unsupported-claim rate Escalation precision	Invented history Wrong severity Missed contraindication	Licensed clinicians and medical safety reviewers
Financial services High-risk, regulated decision support	Credit-policy explanation KYC document review Portfolio-risk summarization	Policy faithfulness Fairness flags Auditability	Proxy discrimination Unsupported advice Incorrect regulatory citation	Risk, compliance, credit, and financial-domain reviewers
Legal High-risk advisory support	Contract review Legal research memo Citation verification	Citation validity Issue spotting Jurisdiction handling	Fabricated case law Missed clause risk Overconfident legal advice	Qualified legal reviewers with jurisdiction-specific expertise
Insurance Claims and underwriting sensitive	Claims summarization Coverage determination support Fraud signal explanation	Coverage-rule accuracy Evidence traceability Escalation quality	Wrong exclusion logic Omitted evidence Unfair claim handling	Claims specialists, underwriters, and compliance reviewers
Customer support Brand, privacy, and escalation sensitive	Answer from policy Refund workflow Escalation classification	First-contact resolution Policy grounding Customer-safe tone	Wrong policy promise PII mishandling Failed escalation	Support operations leads and policy owners
Software engineering Security and reliability sensitive	Code generation Bug triage Security review assistance	Test pass rate Vulnerability introduction rate Patch correctness	Insecure code Broken dependency assumption False-positive review	Senior engineers and application security reviewers
Operations and procurement Process and financial exposure	Vendor comparison RFP analysis SOP assistant	Source coverage Decision traceability Workflow completion	Missed requirement Wrong vendor comparison Unapproved action	Operations, procurement, and finance process owners

SCORING

How scores should be calculated

Each industry score should be computed from a fixed task set, two independent human reviewers, automated regression checks, and cost/latency telemetry captured from the same harness.

The final score should not be a single generic number. It should include task pass rate, groundedness, safety/security pass rate, review burden, latency, and cost per resolved workflow. Models can lead in one industry and fail in another.