Skip to content
Experiment Report
Why does a mismatched call still leave the executor? A collaboration-fidelity probe
Research

Why does a mismatched call still leave the executor? A collaboration-fidelity probe

Six isolated cases distinguish fidelity annotation from an execution veto and constrain causal readings of a negative collaboration result.

2026-11-07中文 →

Why does a mismatched call still leave the executor? A collaboration-fidelity probe ​

“Not executed as issued” can be an audit result or a reason to refuse execution. Those meanings require different program behavior. A system may correctly detect a mismatched tool or argument and still hand the call to the next layer. Reading an audit field as an enforced protection can therefore overstate what a negative collaboration evaluation tested.

This article uses the same-day completed isolated-method probe. In the pinned methods, wrong tool names and arguments were marked unfaithful but still returned as tool calls. Writes in read assignments and truncated responses were blocked. The finding separates observation from constraint in one path. It establishes neither a database write nor the causal source of the full experiment's performance loss.

What did the negative result actually evaluate? ​

The source study compares a single agent with one centralized collaboration implementation. With a fixed simulator, the authors report worse collaboration results in 12 of 13 mechanically qualifying cells and a statistical tie in the other. The primary fit uses 12 closed-model cells, with a weighted mean gain of −28 percentage points and a 95% interval of [−44, −12]. Paper, section 4

The authors also disclose a material validity deviation. A stricter audit found 680 of 2,744 unique write orders executed as issued—about 25%, below the preregistered 90% bar. The planner lacked tool-schema visibility. A small schema-visible check reached 13/18 as-issued writes, but the full grid was not rerun. These are results for that implementation and setting, not a general test of faithfully executed collaboration. Paper, sections 3.4, 4.1 and 5; appendix A

Six cases locate annotation and veto behavior ​

The internal experiment extracts original methods from the content-pinned source and supplies deterministic tool-call responses through a provider double. It calls no model, connects no database and reruns no benchmark. Inputs vary one aspect of a write order at a time:

CaseInternal experimental observationSupported scope
Exact tool and argumentsCall returned; marked as issuedPositive control
Numeric value versus normalized numeric stringCall returned; marked as issuedType-normalization control
Wrong tool nameCall returned; marked unfaithfulNo veto for this mismatch branch
Wrong argument valueCall returned; marked unfaithfulNo veto for this argument mismatch
Write requested in a read assignmentNo call returned; path closedA branch that does block
Truncated provider replyNo call returned; path closedTruncation control

Every result cell is internal evidence, not independent validation. The six-case record retains inputs, actual outputs and limits. Both emission and blocking occur within the same isolated methods, ruling out the explanation that the probe simply returns every call. The narrower finding is that tool and argument deviations are recorded without becoming a veto here.

The public implementation is consistent with that result. Its write path keeps the first tool call, checks the order, sets the executing-write state and returns the call. Write-tagged calls in read assignments are dropped. Tool-schema visibility defaults off, passing no tool definitions to the planner in that configuration. Official execution source

Reliable auditing does not automatically grant veto authority ​

Our interpretation is to record planning, execution fidelity, checker veto authority and external task outcomes separately. The first two support diagnosis. Blocking a mismatch at this point requires a pre-emission constraint. The experiment does not measure how much a veto would improve task scores or establish that stricter rejection would avoid additional inaction.

The paper's single-agent, centralized-collaboration and small single-context plan-review comparisons also vary implementation details such as schema visibility. This probe explains an audit field; it cannot isolate the causal effect of organizational structure for the paper. Exhaustion and inaction also increased. Assigning 74% of average loss to exhausted episodes is the authors' descriptive decomposition, not an intervention establishing exhaustion as the cause of all loss. Paper, sections 4–5

Evidence and sources ​

Paper results and validity deviations remain source-reported claims. Public method branches are inspectable implementation facts. The six deterministic cases are internal experimental evidence. No independent third party validated all results here. Public source, preprint availability and a successful script establish traceability or bounded behavior; they do not establish general collaboration validity or invalidity.

The main sources are arXiv preprint 2610.27655, version 1, sections 3–5 and appendices A–C and the official M1 centralized-collaboration implementation. The source is pinned by the content hash of the snapshot read in this research, not a claim about later branch contents. The internal experiment uses a provider double, does not inspect archived author trajectories and does not independently timestamp the full preregistration history.

What rerun is still needed? ​

A test of architecture beyond this deviation path needs a full task grid with explicit schema visibility, as-issued execution rate, veto rules, budget boundaries and external acceptance. A small set of corrected writes cannot substitute for that rerun. The supported finding is that annotation does not equal blocking. How a real veto changes success, wrongful writes and inaction remains a separate empirical question.

Last updated: