Before putting agents into production: safety guardrails

The difference between a language model producing text and one acting on your behalf is, from an engineering standpoint, a difference in category. In the first case the worst outcome is reading a wrong sentence; in the second, an irreversible operation takes place.

The question organisations ask most in this area is this: what happens if the system gets it wrong? The framework below is the skeleton of our answer.

First, classification: is it reversible?

We sort every tool an agent can use into three boxes.

Read tools. They fetch data and change nothing. These can run freely; the cost of an error is proceeding with wrong information, and that can be caught at the next step.

Reversible write tools. Creating a draft, adding a label, updating a record. If the previous state can be restored after a mistake, the tool belongs in this box. We leave these automatic too, but we retain the state preceding every operation.

Irreversible tools. Sending email, initiating payment, deleting records, writing to an external system. Nothing here runs without human approval. There are no exceptions — the argument that it "usually gets it right" does not apply in this box.

This classification is a business decision, not a technical one. We determine which tool belongs in which box together with the client, and we put it in writing.

Where the approval point goes

The common mistake is placing approval at the very end: the agent runs ten steps and then asks "do you approve?" This renders approval meaningless; nobody verifies a ten-step chain backwards.

Approval goes immediately before the first irreversible operation. Every step before it can run automatically. What the approval screen shows should not be the agent's intent but the operation itself: which address, which content, which amount.

Permission boundaries

The data an agent can reach should be limited to what the task requires. In practice this means the agent connects not to the organisation's entire database but to a narrow view opened for that task. If authority needs to be widened, that is a separate decision — not the agent's own.

The second boundary is time. How long and for how many steps an agent session may run is defined up front. An agent stuck in an infinite loop can cost more than a mistaken one.

Prompt injection

When an agent reads text from outside — an email, a web page, a document — that text may try to give the agent instructions. This is not a theoretical risk; it is something we meet in the field.

No single defence is sufficient here. What we apply: keeping external content in a channel separate from system instructions, constraining tool calls at the schema level, and the human approval that already exists for irreversible operations. The last is the most reliable — because the other two can be circumvented.

Measurement

In agent systems, "accuracy" is not a single number. What we measure:

• End-to-end task completion rate
• Step count on completed tasks (if it is rising, something has broken)
• What proportion of operations sent to approval are rejected (if high, the agent is proposing the wrong things)
• Number of reverted operations

These measurements run on a versioned task set. When the model changes it is measured again with the same set; that is the only way we catch regressions.

Summary

In agent systems the real engineering problem is not the model's intelligence but where authority stops. The whole framework above reduces to one sentence: define in advance the worst thing the system could do, then make it impossible.