Finding the Stage Does Not Prove the Cause
Two agents complete the same task. One follows the intended plan and tool policy. The other obeys an unsafe instruction, crosses the expected control boundary, and happens to deliver the same file.
A terminal evaluator gives both runs the same success. A trace reveals the difference. Yet the richer view still requires restraint: stage-level attribution can locate where an anomaly became visible, but finer labels alone cannot prove root cause, current authorization, or production safety.
The Same Completion Can Mean Different Things
The same-date Research Object examines AgentAudit, a primary study that scores recorded traces across ten lifecycle dimensions: instruction integrity, planning, memory, tool selection, tool invocation, tool correctness, alignment, tool faithfulness, security, and execution integrity.
The experiments cover five models and nine capability or adversarial tasks. The reported results show that models with similar terminal behavior can receive materially different trust classifications. Some runs are not ordinary failures; they are classified as unsafe compliance—the objective is reached through behavior that should not be accepted.
Completed is therefore an outcome fact. It does not say whether policy was followed, the right tool was selected, authorization existed, memory was trustworthy, or execution remained faithful.
A Trace Restores the Process Behind the Outcome
Trace evaluation changes the evidence unit. Instead of asking only whether a result exists, the evaluator can inspect what the agent planned, remembered, selected, invoked, observed, and did next.
A structured lifecycle has two direct benefits. It separates unsafe success from ordinary success, and it associates the first visible anomaly with a stage. Remediation can move from “the model is unreliable” to an inspectable problem: a plan violated a constraint, memory was stale, tool choice was wrong, arguments exceeded scope, or a result was misinterpreted.
A trace can evaluate only captured facts. If an authorization change, external effect, or upstream memory source is missing, the evaluator cannot reconstruct it from absence. A stronger judge cannot reason over evidence it never received.
Three Evidence Layers Cannot Replace One Another
An auditable system needs at least three layers that can reference, but must not rewrite, one another.
The observed-event layer records what happened: event identity, time, actor, model, tool, target, request and result digests, authorization evidence presented at that moment, and external-effect receipts. These are the audited facts.
The evaluation layer records interpretation: evaluator identity, model and version, prompt, examined trace range, per-dimension classification, confidence, calibration examples, and disagreement across evaluators.
The causal and responsibility layer records why and who is accountable. It labels a claim as hypothesis or reviewed finding and preserves supporting and contradicting evidence, missing variables, alternative causes, and independent or human review where required.
AgentAudit directly strengthens the first two layers. The third often requires additional evidence. A planning error may originate in earlier memory contamination. An out-of-scope tool call may follow from an authorization fact that was never shown to the agent. The first visible anomaly is not necessarily the root cause.
Correlated Bias in a Single Judge
The study retains an important limitation: all traces are scored by one fixed judge model, which is also among the evaluated models. This creates correlated interpretation and calibration risk.
More granularity does not automatically create independence. One judge can reproduce the same bias consistently across ten dimensions. Multiple judges can still share training data, framing, or blind spots. If the system emits only a composite trust score, a complex audit may create more persuasive false certainty than a binary outcome.
A governed evaluation should retain evaluator identity and version, test known-positive and known-negative cases, measure false positives and false negatives, and keep disagreement visible. Deterministic rules can anchor some facts, such as whether arguments exceeded a whitelist. Semantic unfaithfulness or unsafe compliance may still require model or human judgment.
What an Auditable Runtime Must Preserve
A useful trace need not retain unlimited context, but it should keep identities that can change the judgment:
- plan version and task constraints;
- provenance of material memory reads and writes;
- tool choice, request parameters, and caller;
- tool response and external-effect receipt;
- authorization or policy version read at action time;
- retry, compensation, handoff, and terminal events;
- the exact event range examined by each evaluator;
- evaluator prompt, model version, classification, and confidence;
- declarations of redacted or missing fields.
Model-generated explanations must remain distinct from tool receipts. The former are interpretations; the latter are external facts. An evaluator may reference an event but should never write its label back as the original event.
Privacy and cost are real constraints. Retention can be scaled by risk, but redaction must preserve enough stable identity to understand who evaluated what. Otherwise the audit becomes a conclusion without an object.
Attribution Should Trigger Investigation, Not Authority
Stage attribution is most valuable when it narrows investigation. If an anomaly first appears in tool selection, reviewers can examine candidate tools, policy, and context. If it appears in memory, they can trace the write source and freshness.
Attribution should not automatically assign organizational blame or grant the next execution. A high trust score is not current business authorization. A low score does not by itself prove that one model is the root cause. The label can trigger human review, more evidence, or bounded repair; a lifecycle transition still needs its own authority.
A terminal report should therefore present outcome and process quality together: whether the task completed, whether execution was faithful, which safety constraints passed, which dimensions remain unknown, and whether evaluators agreed. “Completed” should not consume “how it completed.”
Boundaries and Open Questions
The study sample is limited and does not establish production false-positive or false-negative rates or robust inter-rater agreement. Ten dimensions provide a useful structure, not guaranteed coverage of every long-running failure.
The bounded conclusion is clear: trustworthy evaluation needs trace-level observability and stage attribution, but attribution must retain the identity of an evaluation judgment. Observed events, evaluation conclusions, and causal responsibility are different evidence types; greater granularity does not allow one to replace another.
Open questions include which fields must be immutable facts, how to build domain calibration sets, how to reconcile judge disagreement, what error rate permits automatic remediation, when humans must confirm causal or responsibility claims, and how to balance audit value with sensitive-data protection.
Evidence and source:
- AgentAudit: lifecycle evaluation from execution traces, primary research, 2026.