Skip to content
Experiment Report
When grading writes into the action record
Research

When grading writes into the action record

Six internal substituted-environment probes of pinned original methods show that reward evaluation can append reference actions to the action record, including through a terminal step. Mutation can affect downstream provenance attribution but does not itself establish an incorrect reward; historical prevalence, repair effects and current-version behavior remain unverified.

2026-11-10中文 →

When grading writes into the action record ​

An action list starts empty and gains an action during grading. If a downstream consumer treats the post-grading list as an agent-only trace, a reference action can be assigned to the wrong actor. A pinned, isolated method experiment exposes this mutation mechanism. Terminal-path and empty-reference controls show which provenance questions to ask before reusing the record.

Original methods, synthetic surroundings ​

The probe targets the environment base class of tau-bench at commit 14bf0ef52e595922d597a38f32d3e8c0dce3a8f8. Analysis extracted unchanged syntax-tree method bodies for step handling, data hashing and reward calculation, retaining the original hash functions and omitting the constructor. State, tool semantics and response containers were synthetic substitutes, running under Python 3.12.9. Pinned source

This is internal experimental evidence, not independent reproduction or a complete benchmark installation. It tests how original methods manipulate the supplied action list, not real airline semantics, model interaction or archived trajectories. The six cases and limitations retain that substitution boundary.

What changes, and which controls stop it ​

With no agent actions, direct reward calculation adds one reference set action and returns reward zero. In another case with an already matching target state, the list grows from 1 to 2 with reward 1; evaluating the resulting state again grows it to 3, still with reward 1. Record mutation and reward value are therefore distinct questions.

The terminal path matters too. A finish action sent through the step method enters grading and leaves a list containing finish and the reference set action, with reward 1 and grading information. This confirms an additional call-chain segment beyond directly invoking the reward function, but only in the substituted environment.

An empty-reference control appends nothing and leaves the list length at zero. A nonterminal step returns zero reward, not done, and no grading information. That is an ungraded intermediate response, not evidence of a checked failure. Together, the six cases support mutation on the tested grading paths, not universal list growth or an interpretation of every zero as failure.

Mutation is not itself a wrong reward ​

A benchmark may intentionally replay reference actions to compare state. A growing list alone does not establish an incorrect reward, much less the amount by which a leaderboard should change. The provenance risk arises when a consumer interprets the post-grading list as actions performed only by the agent, mixing evaluator behavior into the actor's record.

A design to test is separate preservation of pre-grading snapshots, action origin and grading events, with repeated-evaluation checks on the original record. Producers need to identify who wrote each action; evaluators need to state whether they operate on a copy; consumers need to identify the record phase they read. Ungraded, checked-failed and unscorable outcomes should also remain distinct. These are checks suggested by the mechanism, not a deployed fix or established cross-system benefit of isolated grading.

Evidence and sources ​

The external context is the evaluation-instrument audit, version 1, together with the audit author's pinned November correction. These explain why action provenance needs scrutiny; they do not turn the author's audit into our new measurements.

The corrected 77/800 flips and 1,596/2,334 provenance count remain author-reported. The older report counted suffix injection, while the correction includes interior injection; the count difference is not a newly observed change in agent behavior. The withdrawn multi-model reordering claim must not be carried forward. We recomputed none of these rates, costs or rankings.

Project code supplies public primary implementation facts; the six substituted-environment cases are internal evidence; downstream attribution risk is our interpretation. Relabeling our own experiment would not make it third-party validation, and neither publication nor citation can serve that role.

Bibliographic record: Silent Instrument Errors Reorder Agent Leaderboards, Yangze Liu, Shandong University; arXiv preprint v1, 3 November 2026. The accompanying correction is published by the audit author and dated 1 November 2026; benchmark environment code is published by the original project, with both sources pinned above. The leaderboard conclusion in the paper title is not a new measurement from this isolated probe.

Which record does a downstream consumer receive? ​

Without the full installation, real airline semantics or archived replay, the probe cannot estimate prevalence. It also does not inspect current upstream versions or recent maintainer responses. The source commit bounds the temporal claim; it cannot establish present-system behavior.

For a team reusing trajectories, a concrete starting point is to pin evaluator and task versions, compare pre- and post-grading lists, and check each action's origin and whether grading occurred. When no pre-grading record exists, reliable provenance recovery remains open. A next test should examine whether record isolation preserves reward semantics and improves audit interpretation before drawing a stronger conclusion.

Last updated: