Capability Is Not Authorization: Least Privilege for AI Agents

Capability Is Not Authorization: Least Privilege for AI Agents

There is a sentence that appears in agent design documents with unnerving regularity: the agent has access to the CRM. It is offered as a scoping statement, as though it settled something. It settles almost nothing.

Can it read one contact, or export the whole database? Can it write to a record, or delete an account? Can it do that to any customer, or only the one in the current conversation? Can it do it a thousand times in a minute? Can it do it at 3am with no human awake? Is there anything it can do that cannot be undone?

Those are six different questions, and a single API key answers all of them with "yes". That gap — between the agent can call this tool and the agent is authorised to perform this specific act, on this specific object, right now — is where most serious agent incidents live. Not in the model. In the permission model that was never written.

The category error

Traditional application security has a mature concept of authorization: a subject performs an action on a resource, and a policy decides. The subject is known, the action is enumerated, the resource is identified, and the decision is made per request.

Agent stacks routinely discard all four properties. The subject becomes a service account rather than the user on whose behalf the agent acts. The action becomes "whatever this tool does", with the tool often being a thin wrapper over a general-purpose API. The resource is not identified in advance because the agent chooses it at runtime. And the decision is made once, at integration time, when someone pasted a token into an environment variable.

The result is a system where the capability granted is enormous and the authorization actually intended is narrow, and nothing in the architecture knows the difference.

The upper path is what most agents run today: one gate, checked once, at build time. The lower path costs more to build and is the only one that produces a bounded system.

Classify tools by what they break, not by what they do

Tool inventories are usually organised by integration — Salesforce tools, email tools, filesystem tools. That taxonomy is useless for risk. Organise instead by the property that determines the consequence of a mistake.

Two things fall out of this table immediately.

The first is that bulk read is a write. An agent that can read one customer record on request and an agent that can pull fifty thousand of them are not in the same risk class, even though they call the same function. Scale converts a benign capability into a data breach, and no permission system that ignores volume will notice.

The second is that "externally visible" is its own class, distinct from irreversible. A message sent to a customer can technically be followed by a correction, but the original cannot be unsent. Retry logic that quietly re-sends on timeout is one of the most reliably embarrassing failure modes in production agent systems, and it is a direct consequence of treating side-effecting calls as idempotent because the code path looked like a normal HTTP request.

Authority must be delegated, not assumed

The most consequential design decision is whose authority the agent is acting under.

If the agent holds its own service credentials, it has a union of every permission any user might need — and the model, at runtime, is choosing which slice of that union to exercise, based on natural-language input from a person whose own permissions the system has forgotten. A support agent asked to "look up the account" will happily look up an account the requesting user has no right to see, because nothing in the chain checked.

The correct posture is delegation: the agent acts with the permissions of the principal on whose behalf it was invoked, and no more. Where that is impossible, the scope must be pinned at session start — this session may touch case 4471, these three documents, this one customer — and enforced at the tool boundary rather than trusted to the prompt.

This point deserves emphasis, because it is where most implementations quietly fail: a constraint stated in the system prompt is not a control. It is a request. The model may follow it, usually will, and cannot be relied upon to. Any restriction that matters must be enforced outside the model, in code that the model cannot talk its way past. If the only thing preventing the agent from deleting production data is a paragraph of prose telling it not to, you do not have a permission model — you have a preference.

Stop rules: the control nobody builds

Least privilege bounds what a single action can do. It does not bound what a sequence of individually-permitted actions can do, and agents are, by construction, sequence generators. Every step can be legitimate and the trajectory still catastrophic: the agent that refunds one customer correctly, then another, then eight hundred, because a retrieval error convinced it that a policy applied broadly.

A stop rule is a pre-declared condition under which the session halts and a human is summoned. Not a warning. A halt.

Useful stop rules, in rough order of how often their absence causes harm:

• Budget exhaustion. Cumulative count, monetary value, or number of distinct objects touched exceeds the session grant.
• Repetition. The same tool call with the same arguments, or a near-identical loop, more than N times — the classic signature of an agent stuck in a cycle it cannot perceive.
• Scope escape. An action targets an object outside the session's declared scope. This should be impossible if enforcement is real; if it fires, something upstream is broken and continuing is not safe.
• Confidence collapse. Repeated tool errors, empty retrievals, or self-corrections cluster together. An agent that is failing to understand its environment should not keep acting in it.
• First-of-kind. The agent attempts an action type it has not performed in this session before, in a class marked externally visible or irreversible.
• Novel instruction source. The agent's plan changes materially right after ingesting untrusted content. That is the observable signature of injection, and it is far easier to detect as a behavioural anomaly than as a string.

Stop rules must be enforced by the harness, must be configured per deployment rather than per model, and must fail closed. An agent that hits a budget ceiling and continues because the check threw an exception is worse than one with no ceiling at all, because someone has been told it is safe.

Evaluating the permission model

You do not learn whether these controls hold by reading the configuration. You learn by trying to break them, deliberately and by a party who did not build them.

That means adversarial scenarios: instructing the agent, through content it ingests rather than through the user turn, to exceed its scope. Constructing tasks whose natural completion path requires an action the agent should refuse. Running sessions long enough for budgets to bind. Checking that a denial is a denial and not a retry with different phrasing.

It also means treating the harness as part of the system under evaluation. Under the SASF framework, tool authority, side-effect control, and stop-rule behaviour are assessed alongside output quality, because an agent that answers beautifully and can delete a production table is not a good system. Findings carry a severity, a root cause, and a remediation, and they land in a versioned Evidence Report — which is the artefact that lets you tell a risk committee what your agent is permitted to do, rather than what you hope it will choose to do.

What to do tomorrow

List every tool your agent can call. Next to each, write the worst single action it permits, and whether you can undo it. Then write the worst thousand actions it permits, because that is the number your rate limit allows.

Anything you cannot undo needs a human on the path today. Anything that scales into a breach needs a volume ceiling today. Everything else needs a budget and a stop rule — and once those exist, the honest test is whether an evaluator who wants your agent to misbehave can make it.