What to Log Around a Tool Call: An Observability Schema for Agents
What to Log Around a Tool Call: An Observability Schema for Agents
Ask a team what they log around a tool call and you will usually hear: the tool name, the arguments, the result, and the latency. That is a reasonable debugging record. It is a poor evidential one.
The difference matters the first time someone asks a question you cannot answer from the log. Why did the agent choose that tool? What did it believe when it decided? Would it do the same thing again? Was the result it acted on the same result the tool actually returned? A schema built for debugging answers none of these, because debugging assumes you can reproduce the failure. With a non-deterministic agent acting on a context you no longer have, you frequently cannot.
The fix is not more logging. It is logging the right things at the right boundary.
The boundary is the decision, not the call
A tool call is the visible artifact of an invisible decision. The agent read a context, formed an intent, selected a tool, constructed arguments, and committed to an action with side effects in the world. Logging only the call captures the last step and discards the reasoning that produced it.
So the unit of record is not `tool_call`. It is the decision: everything the agent had, everything it chose, and everything that happened as a result.
Most schemas instrument C and D. The evidential value is concentrated in A, B, and E.
The schema
Five groups. Each exists because a specific question becomes unanswerable without it.
1. Identity — "which run was this, and where does it sit?"
trace_id stable across the whole task
span_id this decision
parent_span_id the decision that led here
run_id one agent execution
step_index monotonic position in the trajectory
Without `parent_span_id` you have a bag of calls, not a trajectory. The ordering is what lets you reconstruct causation later — which call happened because of which result.
2. Provenance — "what produced this behaviour?"
system_version your deployed agent build
model_id exact model, not the family
model_params temperature, top_p, seed if set
prompt_version the template, versioned
tool_schema_version the contract the model was shown
policy_version the guardrail ruleset in force
This is the group teams skip, and it is the one that hurts most. Six weeks later, "the agent did something strange in April" is unanswerable unless you can say exactly which prompt, model, and toolset were live in April. Provenance is what makes a historical record interpretable rather than merely present.
3. Context — "what did the agent actually know?"
retrieved_doc_ids what came back from retrieval
retrieval_scores and how confident it was
memory_keys_read what prior state was in scope
context_token_count how full the window was
truncated whether anything was dropped
`truncated` is deceptively important. An agent that behaved oddly because half its context was silently evicted is a completely different incident from one that reasoned badly on complete information — and from the outside, the two look identical.
Log document identifiers, not document bodies. You need to know what was in scope, not to duplicate your corpus into your telemetry store.
4. The decision — "what did it choose, and what else could it have?"
tool_name what it picked
tool_args the arguments, redacted per data class
tools_available what it could have picked
authorization the permission that allowed this
`tools_available` is the field almost nobody captures and reviewers always want. "The agent called `delete_record`" is a very different finding depending on whether `delete_record` was one of three options or one of forty. The counterfactual is part of the evidence.
`authorization` records why the call was permitted — the scope, role, or grant that made it legal. Capability and authorization are not the same thing, and the log should not conflate them.
5. Outcome and consequence — "what happened, and what did it change?"
status ok | error | timeout | refused | blocked
result_summary or a hash, per data class
side_effect did this mutate anything external?
reversible can it be undone?
latency_ms
guardrail_actions what fired, what it did
agent_next_action what the agent did with the result
`side_effect` and `reversible` are the two fields that turn a log into a risk record. A read-only lookup that returns garbage is an annoyance. An irreversible external write made on garbage is an incident. Your telemetry should let you separate them with a query, not with an investigation.
Redaction is a schema decision, not a filter
Every field above collides with data protection eventually. Resolve it in the schema rather than at the sink.
Attach a `data_class` to each field and define, once, what is retained per class: full value, redacted value, or hash. Hashing arguments and results is often the right default — it lets you prove that the value the agent acted on was the value the tool returned, and that it has not changed since, without retaining the payload at all. You get integrity without custody of the content.
Decide this before you instrument. Retrofitting redaction onto a schema that assumed full payloads is significantly harder than designing for it.
Why this is an evaluation problem, not just an ops problem
Everything above serves one goal: making an agent's behaviour reviewable by someone who was not there.
That is the same requirement evaluation has. When an Evaluator reviews an agent trajectory under a rubric, they are answering questions the log must support — was the tool choice appropriate given what the agent knew, was the action authorized, was the result used correctly, should this have escalated to a human. A trajectory logged with only `tool_name` and `result` cannot be scored on any of those dimensions. The reviewer is reduced to guessing, and a guess does not belong in an Evidence Report.
Instrumentation quality sets a hard ceiling on evaluation quality. You cannot review what you did not record.
What to do tomorrow
Take one agent trajectory from production. Print every field you currently store for a single tool call. Then try to answer these five questions using only what you printed:
- What else could the agent have called instead?
- Which prompt and model version produced this choice?
- Was any part of its context silently truncated?
- Did this call change anything outside your system, and can you undo it?
- What did the agent do with the result?
Every question you cannot answer is a missing field. Add those fields first — they are, by definition, the ones a reviewer will need.