AI Agent Testing: Turn Claims Into Evidence
Learn how to test AI agent output with real commands, independent review, weighted scoring, and fail-closed security gates. Build a workflow you can trust.
AI Agents Need Tests, Not Self-Approval
An AI agent claims a task is complete. The code looks plausible, the tests sound familiar, and the summary reads with confidence. But confidence isn't evidence - especially when the same context that produced the work also "approves" it. In that setup, review can turn into a polished restatement of the original assumptions.
This is a practical problem, not a philosophical one. A defect can slip through a couple of reassuring rounds and still hit the person who depends on the result. Asking the agent to "be more rigorous" doesn't solve the underlying conflict. Approval has to be separate from the context that generated the work, and it has to verify something observable.
A more reliable approach pairs a builder with an automated critic that runs real commands, plus an independent reviewer. The commands produce measurable evidence; the reviewer checks what automation can't. This article walks through how to set up that loop, score results without hiding gaps, and avoid the false greens that make a testing system seem more trustworthy than it actually is.
Why an agent should not approve its own work
A builder has a particular view of the task. It knows what it meant to implement, which constraints it cared about, and what trade-offs it made. If you ask the same context to review the result, it may focus on whether the output matches that intent. That's not the same thing as checking whether the output actually works.
This isn't that AI can't review code. It can spot defects, suggest tests, and point out suspicious patterns. The problem is narrower: a "review" that relies only on the same context and the same explanation is weak evidence. The model can dress up its assumptions in a more formal tone. A polished review isn't independent just because it's labeled a review.
In real projects, the symptoms look like this:
- The agent keeps saying the work is done, but the review rounds don't converge.
- A defect shows up only after a human runs the project - or a user relies on it.
- A passing summary can't be traced to an actual command or observed result.
- The team can't tell whether a check failed, was skipped, or was never configured.
It's not about whether the agent sounds certain. It's about whether the completion claim is tied to evidence someone else can inspect.
A score is useful only when it has evidence behind it
A numeric score can make a workflow easier to compare, but a score isn't proof by itself. A score of 100 is meaningful only if the checks are relevant, actually ran, and produced output that was kept. If a test was skipped or a command failed silently, the score is just another opinion - this time formatted as a number.
So the goal isn't to chase a green dashboard. The goal is to make the work legible: what was checked, how it was checked, what passed, what failed, and what still hasn't been verified. If a check can't run, that uncertainty belongs in the report. It shouldn't disappear quietly from the calculation.
This same issue shows up outside software. The pattern appears with a data migration, a deployment pipeline, a research summary, or a document used to justify a business decision. In every case, the person (or system) producing the work shouldn't be the only source of assurance.
A practical model for independent AI agent testing
Split the work into four roles. In a small project, one person might juggle multiple roles - but the builder, critic, and reviewer shouldn't be the same unexamined context.
1. Orchestrator: coordinates the work
The orchestrator is the agent or process running the task. It selects the checks, starts each round, helps fix infrastructure issues, and assembles the final report. It shouldn't "invent" a passing result. Its job is to move things forward while preserving the evidence the checks produced.
2. Builder: changes the artifact
The builder writes or updates the code, document, or other deliverable. It receives specific gaps and makes changes to address them. One useful practice is to fix one class of gap at a time. If the build is broken and a security scan finds a secret, mixing both fixes into one vague pass makes it harder to see what changed - and whether the change actually addresses the right issue.
3. Critic: runs checks and records what happened
The critic is a script (or another deterministic mechanism) that executes declared commands, captures their output, and calculates the result. It doesn't ask a model whether the work "looks good." It measures the checks the team agreed mattered.
For example, a project might configure a test suite and a secret scan:
{
"checks": [
{"name": "pytest", "cmd": "set -o pipefail; pytest -q", "weight": 40},
{"name": "no_secret_scan", "cmd": "gitleaks detect --no-banner", "weight": 30}
]
}
Those weights add up to 70, not 100. A real configuration would need another check or adjusted weights. That's intentional: the weights should be validated before the loop starts, not quietly normalized in a way that hides an incomplete rubric.
The critic should keep raw output and report a score plus named gaps. If a command fails, it should produce a visible failure - not a generic "the agent couldn't verify" message.
4. Reviewer: challenges the result independently
The reviewer gets the same artifact and evidence, but starts from a fresh context. It shouldn't inherit the builder's justification as if that were proof. For important work, use reviewers with different areas of focus - for example, one oriented toward quality and one toward security - rather than asking a single general reviewer to cover everything.
The reviewer can reopen the loop when it finds an issue the scripted checks don't cover. This matters because a test suite can confirm that a known behavior still works while saying nothing about whether the implementation is safe, maintainable, or correct for the actual requirement.
Separation doesn't erase bias, and it doesn't magically guarantee correctness. It does make the sources of judgment easier to inspect, and it reduces the chance that one context both creates a claim and "validates" that same claim. For more on making the loop observable, see how a test result becomes evidence.
How to run a measurable test loop
A useful loop is small enough to run repeatedly and strict enough to make a failed check obvious. Start by deciding what the artifact must demonstrate. Then define checks before implementation, run them, address the biggest gap, and get an independent review before delivery.
Declare checks before writing the code
For each check, record:
- A stable name, so the report can identify it.
- The exact command to execute.
- A weight that reflects its importance to this deliverable.
- A precise definition of what counts as a pass.
- A timeout longer than the check's normal runtime.
Weights should sum to 100 if the final score is meant to be a percentage. More importantly, the checks should cover meaningful risks. A set of fast, easy checks is still a weak rubric if it ignores the important behavior of the system.
For a web service, that might mean unit tests, a dependency or secret scan, and an integration check that verifies the expected service identity and response. For a migration, it might mean validation against a representative dataset, an idempotency check, and a rollback test. The details depend on the deliverable; the principle is the same: define evidence before you have a result.
Run commands for real and keep the output
The critic should execute every configured command and preserve its raw output, exit code, and duration. The output should include a score, named gaps, and an overall gate state. For example:
$ python scripts/loop_runner.py --config loop.json --round 3
SCORE=55
GAPS=[pytest::exit_1, no_secret_scan::missing]
GATE=failed
$ echo $?
1
Later, another run might show:
$ python scripts/loop_runner.py --config loop.json --round 7
SCORE=100
GATE=passed
$ echo $?
0
These values matter because they connect decisions to commands and gaps - not because 100 is inherently trustworthy. The report should also state what couldn't be verified. If a test needs a service that was unavailable, that's an unresolved condition, not a pass.
Set a threshold and a round limit
Decide which score is high enough to request independent review, but don't let the threshold override a critical failure. A security gate should block delivery even if the weighted score looks otherwise strong. Set a maximum number of rounds too. Eight is a reasonable example for a bounded workflow; it's not a universal law.
If the loop doesn't converge, investigate the check set, the environment, and the implementation. Repeating the same prompt or lowering the bar isn't a fix. Sometimes the code is wrong; sometimes the tests are brittle or irrelevant. The report should help distinguish those cases.
Review before delivery, then report the limits
The independent reviewer should assess the artifact and the evidence before delivery. Ask for a structured response with explicit findings, severity, and whether each concern could be verified. A strict format helps the orchestrator process the result, but structure isn't a substitute for judgment.
The final report should answer four questions:
- What was checked?
- What passed and failed, with raw evidence available?
- What did the independent review find?
- What was not verifiable?
That definition of "done" is more useful than a confident completion message. If you're relying on a passing gate, also review the failure modes that create false greens.
How a green result can still be wrong
Automation can produce false confidence when the test harness is flawed. A green result only says that, as configured and executed, the checks passed. It doesn't guarantee that the right checks ran against the right target.
A pipeline hides the failing exit code
In some shells, a command like pytest -q | tail returns the exit code of tail, not the test runner. The output can look fine even while the tests failed. Use set -o pipefail where supported, or configure the shell so pipeline failures propagate correctly. Verify this behavior in the environment where the critic runs.
The timeout hides the actual failure
A timeout shorter than a check's real duration can terminate the critic before it writes a useful result. Then the orchestrator may see an empty log or an unclear status and handle it incorrectly. Set timeouts based on observed runtime, preserve logs even when a process is terminated, and treat timeouts as a failure or inconclusive state - not as a pass.
The check reaches the wrong service
An HTTP 200 only proves that something responded. A port collision or stale process can make a check talk to the wrong application. Confirm identity using a known version, title, payload, or other expected signal. Reachability and correctness are different checks.
A baseline is changed just to make the test pass
Visual snapshots and other baselines can go stale. Updating them can make sense when an intentional product change alters the expected result. But regenerating them just to silence a failing check erases evidence of a regression. Baseline changes should be reviewed, justified, and recorded as deliberate changes.
A weighted score masks a critical issue
Weights can summarize a set of checks, but averaging isn't always appropriate. A serious security finding shouldn't be offset by a pile of passing style or unit checks. Define hard gates for issues like exposed secrets, high-severity vulnerabilities, or authentication and injection flaws. Treat an inconclusive security result as blocked until it's resolved.
This is a fail-closed rule: when a required safety condition can't be verified, delivery stops. It's stricter than a score threshold, and that's the point. The cost of blocking briefly is usually lower than the cost of shipping a known or unresolved critical risk.
Make completion claims inspectable
The practical shift is simple: stop treating an agent's statement that something works as the result. Define checks, run them outside the builder's judgment, retain the evidence, and use a fresh reviewer to challenge what automation can't establish.
A score isn't the goal. A reliable conversation about what passed, what failed, and what remains unknown is. Start with a small set of checks tied to real risks, add hard security gates where needed, and keep the loop bounded. If the workflow can't explain why it passed, it isn't ready to be trusted.
For implementation details and the original AI Test Loop package referenced in the source, see the AI Test Loop project. Use it as a starting point, then adapt the checks to your own system and delivery risks.