Most High-Risk AI Is Not a Chatbot: Evaluating the Whole Model Stack
Most High-Risk AI Isn't a Chatbot
The public conversation about AI risk is a conversation about chatbots. Hallucinations, jailbreaks, prompt injection — all real, all important, all about large language models. But step inside a bank, a hospital, or an insurer, and the AI making the consequential decisions is usually not generative at all.
A gradient-boosting model decides who gets credit. A convolutional network reads the radiology scan. A graph neural network predicts the drug interaction. A time-series model flags the adverse event. A classical classifier screens the job applicants. Under the EU AI Act's Annex III, these are the systems that count as high-risk — and most of them have never seen a transformer.
This matters because these systems fail in ways no LLM benchmark measures. A credit model trained on metropolitan borrowers quietly underperforms on rural applicants. A medical-imaging classifier trained on one demographic fails silently on another. A model that says "85% probability" is wrong about how often it's right. None of these failures look like a hallucination, and none of them surface on a leaderboard.
The model stack you actually have to evaluate
A complete evaluation programme covers far more than conversational AI. Each architecture has distinct failure modes and demands a distinct method.
| System type | Where it's used | What actually fails |
|---|---|---|
| LLM / RAG / agents | Copilots, assistants, support | Fabrication, jailbreaks, tool misuse |
| Tabular / structured ML | Credit, fraud, underwriting | Distributional shift, calibration, proxy bias |
| Computer-vision classifiers | Medical imaging, inspection | Misidentification, adversarial perturbation |
| Time-series / anomaly | Adverse-event, monitoring | Drift, threshold misconfiguration |
| Graph neural networks | Drug interaction, networks | Structural attacks, representation gaps |
| Voice agents | Claims, triage | Accent bias, failed escalation |
| Memory-augmented systems | Anything with persistent memory | Confabulation, cross-user leakage, stale recall |
For a language model, evaluation means scoring outputs against expert-written answers. For a classical model, it means quantitative analysis — discrimination (AUC, Gini), calibration (ECE, reliability curves), fairness (the four-fifths rule, equalized odds), and drift (Population Stability Index). For a persistent-memory system, it means multi-session protocols that test whether the system correctly remembers and correctly forgets. One methodology cannot serve all three.
Six layers, one verdict
The SingleAxis Standardized AI Safety Framework (SASF) evaluates any of these systems across six independent layers, wrapped by a governance meta-layer. A system has to pass every applicable layer — and Safety can fail the whole evaluation on its own.
Underneath sits a structured taxonomy — 16 failure categories and 162 codes spanning everything from fabrication and bias to model calibration, pipeline drift, long-context decay, and persistent-memory confabulation. Every finding is classified to a code, every code maps to a regulation, and every verdict is computed from scored evidence, not impression. Two evaluators score each task independently; agreement is measured with Cohen's Kappa (minimum 0.70); disagreements on critical dimensions go to a third adjudicator.
The frontier: systems that remember
The newest failure surface is memory. As enterprise AI systems ship persistent memory — retaining information across sessions — they introduce risks that short-context testing cannot see: a system that confabulates a memory the user never created, leaks one user's memory into another's session, or acts on stale information after the facts have changed. A financial advisor acting on a year-old client profile, or a clinical assistant that "remembers" an allergy that was never mentioned, is a new category of harm. Evaluating it requires testing across sessions, not just across turns.
Why this has to be human-led — and documented
Automated tools are essential for scale, regression, and drift monitoring. But they cannot judge whether a clinical recommendation is contextually appropriate, whether a credit model's explanation reveals proxy discrimination, or whether a refusal is proportionate. Those are judgment calls — and regulators require a named human to make them. The EU AI Act's human-oversight and testing obligations for high-risk systems now apply from 2 December 2027 following the Digital Omnibus deferral, and they ask for documented evidence, not benchmark screenshots.
That evidence is the point. The output of a SASF evaluation is an Evidence Report: a structured, auditable record — verdicts by layer, findings by severity, metrics disaggregated by group, mapped explicitly to the regulations that govern the system. It is built to drop straight into a conformity-assessment file.
The era of evaluating AI as if it were all chatbots is over. The systems making the highest-stakes decisions are the ones least likely to be generative — and the only defensible way to deploy them is to evaluate the whole stack, by qualified people, with proof. See how that works for systems you've already built.