Reliable agents need operating boundaries

The demo leaves out the hardest question

A demonstration usually asks whether an agent can finish a task. An operating team needs to know what happens when it cannot. The difference becomes visible when a dependency is unavailable, a document contradicts an earlier instruction, or the same action is requested twice. A persuasive final message cannot resolve any of those conditions. The application must know which changes actually happened and which remain uncertain.

Anthropic's engineering guidance distinguishes predefined workflows from systems where models direct their own tool use. Separately, Microsoft's research on multi-turn conversations documents reliability problems when requirements arrive across several turns. These sources support a practical starting point: evaluate the full interaction and its environment. They do not establish that every current model will fail in the same way or that a particular architecture eliminates the problem.

Write an operating contract before a prompt

For each workflow, define the intended outcome, the resources the agent can touch, and the conditions that end the run. Consider a hypothetical assistant preparing a customer account update. Its contract might permit reading approved records and drafting a proposed change, while requiring a separate authorization step before the update is committed. Success means the right record changed with the right values, not that the agent reported completion.

Keep the contract small enough to test. Include a maximum elapsed time, a budget for tool calls, and a clear escalation destination. These limits belong in application code because a natural-language request to stop after a certain point is not an enforcement mechanism. When the limit is reached, preserve the work and explain what remains. A bounded partial result can be more useful than another speculative attempt.

Make uncertainty a real state

Many systems model work as pending, running, or complete. Add states for waiting on evidence, awaiting approval, and an external result that is unknown. Unknown is especially important after a network timeout: the remote service may have accepted the action even though the response never reached the agent. Retrying blindly can turn a temporary communication problem into duplicate work.

Use the external system's record or a stable operation identifier to reconcile the outcome. Keep observations separate from the model's interpretation of them. An API response, a verified document version, and a generated summary serve different purposes. When a summary says that a submission succeeded but the operation record contains no confirmation, the system should expose that inconsistency instead of choosing the more reassuring answer.

Test the boundaries you intend to trust

A useful reliability exercise starts with an ordinary task and changes one environmental condition at a time. Remove a permission halfway through the run. Return an empty result from a search. Make a tool time out immediately after accepting a write. Supply two records with similar names. Check whether the system reaches a safe, understandable outcome and whether a person can continue without reconstructing everything.

Measure completion alongside the cost of recovery. An agent that completes slightly fewer tasks but leaves clear evidence and recoverable state may be the better choice for a small operations team. Record how many runs require intervention, how long intervention takes, and which failures recur. Avoid interpreting a single aggregate success percentage as proof that all categories of work are equally dependable.

Assign a named owner to each exception category. An unclear record match may belong to operations, while a rejected credential belongs to the integration owner. Route the evidence with the task so the recipient can act immediately. Without this ownership map, a safe stop merely moves unfinished work into an unmonitored queue and makes the automation appear more successful than it is.

Expand autonomy through evidence

Begin with a narrow action surface and one clearly owned workflow. Let the system prepare work, observe its mistakes, and establish a representative evaluation set. Increase its authority when evidence supports a specific expansion, such as allowing a reversible update within a defined scope. Do not treat success at drafting as evidence that the same system can safely execute every downstream action.

The architectural question is where judgment helps and where certainty is required. Models can interpret incomplete requests, compare evidence, and propose next steps. Conventional software can enforce identity, limits, valid transitions, and recorded outcomes. Metrosion's editorial position is that dependable AI comes from making those responsibilities explicit. The most useful milestone is a workflow that people can understand, interrupt, and recover when conditions change.