AI Agent Reliability: Build Systems That Hold Up

State is part of the agent, not an implementation detail

A multi-step task needs a record of what has happened so far. If the process stops after a tool call, the system needs to know whether the call completed, what it returned, and whether the next step is actually safe to run. If that information only lives in a transient process or an in-memory chat, it's fragile. A restart, timeout, or deployment can wipe out the context you need to recover correctly.

When you persist each completed step, you create a dependable boundary. The system can resume from known results instead of asking the model to reconstruct its history from a long transcript. It also makes incidents easier to investigate: operators can see the sequence of decisions and outcomes rather than guessing from logs after the fact.

Evaluation is regression testing with a different surface area

A team described running the same forty cases whenever it changed a model or prompt, and treating regressions as a reason to block a release. The point isn't that forty is magic. The point is the discipline: keep a repeatable set of representative tasks and check whether a change makes the system worse.

Agent behavior is variable, so a test suite won't prove every future run is safe. But it can still catch common regressions: a prompt that stops asking for a required field, a model update that calls the wrong tool more often, or a formatting change that breaks downstream processing. Without a baseline, teams tend to lean on a few successful demos - and hope.

Tool calls deserve ordinary API discipline

A model-facing tool is still an API boundary. It needs a clear input schema, predictable error responses, and a policy for whether a request can be retried. This matters because the model can misunderstand what a tool does or call it with incomplete information.

The broader lesson is pragmatic: keep the parts that are already well understood. Use durable storage for progress, regression tests for behavior, and explicit contracts for tools. The AI component may introduce uncertainty, but it doesn't replace the need for engineering practices that contain that uncertainty.

Model the workflow as explicit states

Start by listing the meaningful states in the task. A support workflow, for example, might move through received, classified, account_checked, action_proposed, approved, and completed. Then define which transitions are allowed, and what conditions permit each transition.

You don't have to build a heavy workflow platform to do this. A small state machine can be a table, a set of typed transitions, or a few carefully named functions. What you're buying is inspectability. A reviewer should be able to ask what happens if account lookup times out or approval is declined - and the answer should be part of the design, not hidden in a prompt.

Separate model decisions from system guarantees

The model can suggest a classification or propose an action. The application should enforce permissions, validate parameters, and decide whether the action is allowed. Treating the model's natural-language response as authorization is a category error: "confidence" in the response doesn't establish that the action is safe or permitted.

For each transition, decide what must be true before the workflow advances. For example:

• A required field is present and passes schema validation.
• The user's permissions allow the proposed operation.
• Any required human approval has been recorded.
• The previous step's result has been persisted.
• The next action fits the workflow's retry and risk policy.

These checks make failures legible. Instead of a vague "agent failed" message, the system can report that a required value was missing or that approval didn't happen.

Make retries an explicit decision

Retries are not automatically safe. Repeating a read-only lookup is usually harmless. Repeating a payment, account update, or outbound message can create duplicate effects.

So the retry policy should be attached to the operation - not applied blindly to the entire agent loop.

When a step fails, the system needs to distinguish between "the operation did not happen" and "the system doesn't know whether it happened." The second case is riskier. It may require checking the downstream system before trying again. A visible state machine turns that uncertainty into a design problem you can address, rather than a surprise hidden in orchestration code.

Build tests around real workflow outcomes

A compact evaluation set should represent what the system is expected to do, covering normal cases and important edge cases. Include incomplete requests, ambiguous instructions, unexpected tool errors, and scenarios that should trigger a refusal or escalation.

For each case, write down what "acceptable behavior" looks like. In some tasks there's one correct answer. In others, multiple answers might be fine - but a boundary still exists. For instance, the agent might choose any reasonable category, but it must not issue a refund without the required authorization.

Run the same cases after changes to the model, prompt, tools, or workflow. When outputs vary, repeat runs or score them against explicit criteria instead of assuming that a single successful result proves stability. A test suite is a regression signal, not a guarantee.

Measure whether errors reach the outside world

Track both model behavior and operational outcomes. Depending on the use case, measures might include:

• The proportion of tasks completed without human correction.
• The rate of invalid or unavailable tool calls.
• The number of actions blocked by validation or permission checks.
• The percentage of uncertain cases that were correctly escalated.
• The rate of duplicate or irreversible actions.
• How often humans get involved, and the cost of that involvement.

Context matters. A high escalation rate could mean the system is cautious in the right places. Or it could mean the system struggles with routine work. Pair the metrics with reviews of actual cases and the operational cost of the behavior.

Use risk-weighted release gates

Not every regression should block every release. A small change in wording might be fine. A change that permits an unauthorized transaction is not.

Set release thresholds based on consequence. For high-impact actions, even a small number of severe failures might justify blocking a change, even if average task accuracy improves.

This approach makes trade-offs explicit. It also avoids a common evaluation mistake: optimizing a single score while ignoring what happens at the boundaries. The "right" benchmark is the one that helps you make a deployment decision - not the one that produces the biggest percentage.

Make tool contracts explicit

Each tool should have a narrow purpose, a defined input schema, and predictable responses for success and failure. Avoid a single catch-all tool that performs unrelated operations based on loosely interpreted text. Narrow tools are easier to validate, authorize, test, and audit.

Errors should tell the application what happened without forcing the model to infer it from prose. For example, distinguish invalid input, a permission denial, a temporary service failure, and an unknown result caused by a timeout. Different cases should lead to different next steps.

Give side effects an idempotency key

An idempotency key identifies one intended operation, so a retry doesn't accidentally perform the same action twice. Generate a key for the logical action, pass it to the downstream service where supported, and store it alongside the workflow state.

This matters when a request times out after the remote service may have already completed the operation. A fresh request with a new identifier could create a duplicate. Reusing the original key gives the receiving system a chance to return the existing result instead of applying the effect again.

Idempotency isn't universal. The downstream service has to honor the key, and the application has to define when two requests count as the same operation. If your system can't support idempotency, the safer recovery might be to query for the outcome or require human review rather than blindly retry.

Persist before advancing

After a tool call returns, store the result before moving on. That creates a durable checkpoint. If the worker crashes later, it can resume from the recorded result instead of repeating the call or asking the model to guess what happened.

A practical production checklist is:

• Persist the workflow state and each completed step.
• Assign an idempotency key to every side-effecting operation.
• Define retry behavior for each tool, including ambiguous timeouts.
• Validate tool inputs and enforce authorization outside the model.
• Record enough detail to reconstruct a run without storing unnecessary sensitive data.
• Require human review when the impact of an error exceeds the value of full automation.

The right level of controls depends on the boundary of harm. An internal research assistant that produces a draft has a different risk profile than an agent that changes a customer account. Start at the boundary of harm, then add controls that reduce real failure modes.