Skip to content
Comparative Study
A completed task still needs separate evidence ledgers
Research

A completed task still needs separate evidence ledgers

Specification counterexamples show why outcomes, procedure and measurement support cannot be collapsed into an unqualified success rate.

2026-11-05中文 →

A completed task still needs separate evidence ledgers ​

A result can be complete while the procedure that produced it remains unacceptable. In BABA-Bench, the authors report 921 outcome-complete runs but only 667 strict passes among 1,974 runs. The 254-run difference is about 27.6% of the outcome-complete group, rather than 27.6% of all runs. This is an author-reported result, not a reproduction by this Research Center. [1]

The practical question is what an evaluation loses when it collapses completion, procedural validity and measurement availability into one success number. Comparing BABA-Bench with MedHarnessBench suggests that these observations need separate identities. Our same-day internal specification probe makes the distinction explicit without claiming to reproduce either benchmark.

The same outcome can carry different procedural evidence ​

BABA-Bench defines strict pass as a conjunction: required outcomes must be reached, necessary permissions must exist, actions and messages must be grounded in evidence, and prohibited actions must be absent. The relevant permission and evidence belong to the action when it occurred. A favorable final state cannot reconstruct those missing conditions. The paper also excludes its 134 transcript-conduct rules from strict pass; even this stricter result is not an exhaustive compliance certificate. [1, sections 3.2 and 6]

Our internal probe encoded that four-condition definition as an executable Boolean specification. Across all 16 configurations, eight have terminal success and only one has strict success. Seven terminal-success configurations therefore fail the conjunction. These are enumerated configurations, not sampled agent behavior: 7/8 is not an observed failure rate. The probe establishes the information discarded by a terminal-only predicate, not whether the published implementation realizes every rule correctly. [3]

The authors' empirical failure categories also overlap: a run can lack permission, lack grounding and contain a prohibited action. Adding those category counts would double-count some runs. This matters when a team tries to turn a benchmark table into a remediation queue: a distinct run and a distinct failure mechanism are different counting units. [1, section 4.2]

MedHarnessBench uses a different evidence design. Its process dimensions retain unavailable values when a trajectory does not support measurement, and aggregates are reported with their applicable support. Current pinned runner documentation also distinguishes the harness gate from the environment's native outcome. Those definitions are observable source facts; they do not establish that all reported executions were scored correctly. [2, section 3.2.2; 4]

Missing measurements can change the ranking without changing behavior ​

Consider the constructed internal example below. Entries are illustrative scores, not measurements from either paper. A dash means unavailable, rather than failed. [3]

Constructed sampleThree process entriesMean over available entriesMean after filling unavailable entries with zero
A0.9, unavailable, unavailable0.9 on 1 entry0.3 on 3 entries
B0.7, 0.7, 0.70.7 on 3 entries0.7 on 3 entries

The supported means place A above B; zero filling reverses that ordering. Nothing about the underlying behavior changed. The denominator and the meaning assigned to absent evidence changed. Filling with one instead would raise A to approximately 0.967, illustrating the opposite distortion. The internal example establishes neither the prevalence of missingness nor bias in either published benchmark. It also does not establish that A is better: its apparent advantage rests on much narrower measurement support.

Our interpretation is that support should travel with a score, rather than be hidden inside an imputation policy. A comparison needs to state what was measurable, on which trajectories, and with what denominator. When support differs, reporting both the supported value and coverage preserves a question that a scalar ranking would erase.

Three attempts can measure opposite notions of reliability ​

BABA-Bench's all-three consistency measure requires success in every one of three seeds. MedHarnessBench's reported Pass@3 uses a plug-in estimate of at-least-one success, 1 - (1 - p)^3, from three trials, then macro-averages across tasks. Similar names do not make these estimands interchangeable. [1; 2, section 4]

For the internal constructed outcomes [1, 1, 0], the observed all-three indicator is zero. With the empirical rate p = 2/3, the plug-in at-least-one value is 26/27, approximately 0.963. Neither value contradicts the other: one asks about consistency across all attempts, the other estimates obtaining a success with repeated opportunities. The calculation is not a claim that the trials are independent in practice, that the estimate is precise, or that the two systems can be ranked against each other. [3]

Budget semantics introduce a further boundary. The pinned native-baseline protocol notes that a particular recovery arm may have an additional action reserve. Inspecting that field is necessary before asserting equal total action cost. This documentation observation does not establish that budgets in the original paper were unequal. [5]

What a completion record should preserve ​

These distinctions suggest a record with separately inspectable outcome, action-time permission and grounding, process-score support, and intervention attribution. That is our engineering interpretation of the comparison, rather than a new benchmark result.

Intervention attribution needs its own chain. MedHarnessBench distinguishes eligibility, firing, acquiring control, delivery, uptake and behavior. A trigger count cannot establish that an intervention reached an agent, and delivery alone cannot establish that it changed the result. A completion record should retain the relevant step and its observation basis before assigning credit to a runtime mechanism. [2, section 3.2.3]

The useful next comparison would align success definitions, support denominators and budgets before examining paired outcomes. It would also distinguish source-defined predicates from an independently audited scorer. In BABA-Bench, the translator is itself an evaluated model and its accuracy on evaluated trajectories was not measured. In MedHarnessBench, the inspected implementation exists, but exact paper-to-code binding and complete reproduction remain unresolved. We executed neither benchmark engine nor model, clinical environment or judge. The internal probe is a definition-level counterexample, not clinical evidence or independent validation. Publication status and available code establish provenance and inspectability; neither establishes general validity. [1–3]

The remaining uncertainty is operational: can long-running systems retain these evidence identities through retries and partial measurements without silently changing the question their final success number answers?

Evidence and sources ​

Public definitions and pinned documentation describe what their authors specify. Empirical paper numbers remain author-reported claims. Our constructed calculations are internal evidence from 2026-11-05, with no independent reproduction established. The proposed completion-record practice is our interpretation.

  1. BABA-Bench: Evaluating Long-Horizon Patient-Advocacy Agents, paper authors, arXiv preprint v1, October 2026; retained primary text read on 2026-11-05, especially sections 3.2, 4.2 and 6.
  2. MedHarnessBench: A Unified Benchmark for Healthcare Agent Evaluation, paper authors, arXiv preprint v1, October 2026; retained primary text read on 2026-11-05, especially sections 3.2 and 4.
  3. Internal specification and synthetic-counterexample record, Research Center, measured 2026-11-05; extracted from the admitted Analysis experiment, not rerun during writing.
  4. Runner scoring documentation, MedHarness repository maintainers, pinned implementation documentation; read 2026-11-05.
  5. Native baseline comparison protocol, same maintainers and pinned revision; read 2026-11-05.

Last updated: