Red-Teaming LLM Applications: A Practical Attack Taxonomy

Red-Teaming LLM Applications: A Practical Attack Taxonomy

Most LLM red-teaming exercises look the same. A few smart engineers spend a Friday trying to make the assistant say something embarrassing. They succeed, because they always succeed. Screenshots are pasted into a channel, a system prompt is hardened, and everyone moves on believing the system has been tested.

It has not been tested. It has been poked. The difference is coverage: a test is something you can fail systematically, repeat later, and report against. An anecdote is something you can only tell.

Red-teaming becomes evidence when it is structured — when attacks are organised into a taxonomy, the taxonomy is mapped to a test matrix, and each cell of that matrix is exercised with a defined pass condition. This briefing sets out the taxonomy we find holds up across LLM applications, and how to turn it into a matrix you can actually run.

Attack the application, not the model

The first correction most teams need: you are not red-teaming a model. The model was red-teamed by its provider against generic harms — weapons, self-harm, hate. Repeating that work tells you almost nothing about your risk.

Your risk lives in the application: the system prompt, the retrieval corpus, the tools, the memory, the downstream consumers of the output. A model that flatly refuses to explain how to build a weapon will cheerfully read a poisoned document and email your customer list to an attacker, because nothing in its safety training says otherwise. The harm is not in the tokens. It is in what the tokens cause your infrastructure to do.

So the unit of red-teaming is the deployed surface, with its real tools, its real corpus, and its real permissions.

The taxonomy

Five families cover the overwhelming majority of what we see. They are distinguished by what the attacker controls and what they are trying to move.

A. Instruction hijack. The attacker's text is treated as instruction rather than data. Direct injection is the naive case — a user typing "ignore your instructions". The dangerous case is indirect: the instruction arrives inside content the system ingests without a human ever typing it. A support ticket. A resume. A web page the agent fetches. A cell in an uploaded spreadsheet. The model has no reliable way to distinguish the developer's instructions from a sentence in a retrieved document, because both arrive as tokens in the same context window.

B. Guardrail evasion. Classic jailbreaking: getting the model to emit content it was trained to withhold. Role framing ("you are an actor playing..."), hypothetical distancing, encoding (base64, leetspeak, homoglyphs), translation into a lower-resource language, splitting the payload across turns so no single message trips a filter. Worth testing, but for most enterprise applications this family is less consequential than it looks — the reputational blast radius of a rude chatbot is smaller than the operational blast radius of a hijacked agent.

C. Data exfiltration. The attacker extracts what the model can see. System prompt extraction is the entry point and usually succeeds eventually; assume your system prompt is public and design accordingly. The serious variants are cross-tenant retrieval (a query that pulls chunks belonging to another customer because the vector filter was applied after retrieval rather than during it) and exfiltration through side channels — a markdown image whose URL embeds the data the model just read, rendered by a client that dutifully fetches it.

D. Tool and action abuse. Where agents turn a content problem into an incident. The model is persuaded to call a legitimate tool with attacker-chosen arguments: a database query with a widened predicate, an email send to an external address, a file write outside the sandbox. The confused-deputy pattern is the canonical form — the agent holds credentials the attacker does not, and the attacker simply asks the agent to use them.

E. Availability and cost. Underrated and easy to demonstrate. A prompt that induces a very long output, a document that triggers a retrieval loop, a task that causes an agent to call an expensive tool repeatedly. There is no content harm at all; there is a bill and a saturated queue.

From taxonomy to test matrix

A taxonomy is a filing system. A test matrix is what you run. Build it by crossing attack families against your actual injection surfaces — every channel through which attacker-influenced text can reach the context window.

Each populated cell is a test. Each test needs three things written down before you run it: the payload, the observable pass condition (not "the model behaved" but "no outbound network call was made to a non-allowlisted host"), and the surface it was delivered through. Without the third, you cannot tell whether your defence generalises or whether you only ever tested the front door.

Two rules keep the matrix honest.

Score at the effect, not the utterance. A model that refuses in prose but still emits a tool call has failed. A model that produces a compliant-looking answer while the retrieval layer silently served another tenant's document has failed. Instrument the tool calls, the retrieval IDs, and the outbound requests, and evaluate those — the transcript is the least reliable artefact in the system.

Fix the class, not the string. When a payload works, the remediation is not to add that string to a blocklist. It is to identify the property that made it work — that retrieved content is trusted, that a tool accepts an unvalidated recipient — and close it. Then re-run the whole row, because a class fix should move a row, not a cell.

What a red-team result must record

Screenshots do not survive contact with an auditor. A defensible adversarial test produces a record that another qualified person could reproduce and disagree with: the exact payload and delivery surface, the system configuration at the time (model version, system prompt version, tool schemas, corpus snapshot), the observed effect at the instrumented layer, a severity that reflects consequence rather than novelty, and a root cause naming the architectural property at fault.

That is also the difference between a red-team exercise and an evaluation. Under a structured framework, adversarial findings are scored against a rubric with anchored severity, disagreements between Evaluators on whether a case is a real failure are surfaced and adjudicated rather than argued in a thread, and the result lands in a versioned Evidence Report you can hand to a regulator. A screenshot proves someone found something once. A record proves what your system does, under what configuration, against a defined attack set.

What to do tomorrow

List every channel through which text you do not control can reach the context window — user input, uploads, retrieval, tool responses, memory. That list is your column headers, and it is almost always longer than the team expects.

Then take one row. Take D, tool abuse, because it has the shortest path to a real incident. Write five payloads, deliver each through every surface in your list, and instrument the tool-call layer so you are scoring what the system did rather than what it said. Run it against production configuration, not a hardened test rig.

You will fail some cells. That is the point. A red-team exercise that finds nothing has not demonstrated safety — it has demonstrated that you did not try the surfaces that matter.