When an AI Agent Checks Its Own Evidence: How to Build Truly Independent Evaluation

Imagine an AI agent is asked to fix a page on a website. When the work is finished, the same agent writes a report saying that the change was completed correctly. What happens if the evaluation system treats that report as evidence instead of checking the real page?

The process looks professional: the task is complete, a report exists, and the quality gate is green. In reality, however, the system has only confirmed its own claim—not the result the user can actually see.

We can call this evidence contamination: the evidence source depends on the executor or evaluator, so the test is no longer sufficiently independent.

Why is evaluating an AI agent different from checking a simple answer?

An agent usually does more than generate text. It calls tools, changes files, operates user interfaces, alters system state, and makes decisions across several steps. A final response alone therefore cannot prove that the work was completed correctly.

Anthropic's guidance on agent evaluations recommends examining the full execution trace: outputs, tool calls, intermediate results, and environmental changes. It also emphasizes using a mix of deterministic, model-based, and human evaluators depending on the task.

OpenAI similarly presents trace grading as an end-to-end way to evaluate workflows, because a failure may happen in an intermediate step rather than in the final answer.

Where does the problem begin?

A vulnerable architecture often follows this path:

1. The agent performs the task.
2. The agent writes a report about its own work.
3. The evaluator reads that report.
4. The presence of expected words, images, or statuses is treated as a pass.

In this design, the evaluator may be testing whether the report looks correct rather than whether the product is correct. If the report is incomplete, stale, or overly optimistic, the quality gate inherits the same error.

The three layers that must be separated

1. Executor

The executor is the agent or service that performs the change. Its output may be code, a file, a configuration change, or an action completed on a platform.

2. Evidence collector

This layer must collect evidence directly from the real source: the final file, an API response, the live page, database state, a trustworthy log, or a screenshot of the interface. Evidence should not be derived only from the executor's explanation.

3. Evaluator

The evaluator compares independent evidence with predefined criteria. It may use deterministic tests, mechanical rules, an LLM, or human review—but it should not be the primary source of the same evidence it is judging.

A practical pattern for independent evaluation

Before executing any task, define four things:

• Observable outcome: What exactly must be true when the task is complete?
• Source of truth: Which file, API, page, or record can answer that question?
• Evaluation method: Which facts require deterministic tests, and which require model or human judgment?
• Stop condition: When must the agent refuse to mark the task as done?

After execution, preserve this order:

Execute → Collect independent evidence → Grade → Human review when needed → Accept or return

For example, if the task changes a web page, the agent's explanation is not final evidence. The system should open the real build, locate the target element, test its behavior, and compare the result with the acceptance criteria.

Can an LLM be the evaluator?

Yes—but not without limits. An LLM is useful for open-ended qualities such as clarity, tone, completeness, or alignment with a narrow rubric. Deterministic facts such as an HTTP status, file existence, field value, test result, or identifier match are better checked with mechanical rules.

A safer pattern combines:

• deterministic tests for measurable facts;
• an LLM for qualitative judgment under a constrained rubric;
• a human for high-risk decisions, public publication, spending, or access changes.

Five signs of evidence contamination

Recheck the evidence source if you notice any of these signs:

1. The report always passes even when the visual output is clearly wrong.
2. The only evidence appears in a comment or response written by the same agent.
3. The evidence does not identify the task, file, or version being evaluated.
4. The evaluator cannot explain which source of truth produced the result.
5. Repeating the test in a clean environment produces a different outcome.

The main lesson

Independent evaluation does not simply mean using a different model. If two different models read the same contaminated report, the evidence is still not independent. Real independence comes from separating roles and data sources.

A trustworthy AI agent is not a system that merely reports well. It is a system whose claims can be checked again against an independent source.

Before turning the next gate green, ask: Are we checking the product—or only the agent's report about the product?

Sources

• Anthropic — Demystifying evals for AI agents
• OpenAI — Introducing AgentKit and trace grading