Skip to content
Experiment Report
Stricter acceptance, complete coverage? A completion-contract probe
Research

Stricter acceptance, complete coverage? A completion-contract probe

Nine synthetic controls distinguish stronger assertions from broader coverage: exact named-tool order and declared state predicates retain their own omission boundaries.

2026-10-29中文 →

Stricter acceptance, complete coverage? A completion-contract probe ​

One checkout and two checkouts can receive the same completion verdict. In an internal experiment, the same grading helpers received two synthetic traces: one checkout with a balance of 155, and two checkouts with a balance of 310. A contract requiring the checkout action and subsequence order, with no balance assertion, accepted both. The verdict described its declared conditions accurately; it did not establish that the business effect occurred once.

This article uses the nine controls already completed in the same-day Research Object. No new experiment is introduced. Its proposition is narrow: even a strict completion verdict is bounded by its declared checks. For digital-employee acceptance, how a check evaluates and which consequences it covers need separate review.

Two passing records, different balances ​

CRM-Agent-Bench combines declared action, argument, order, response, final-state and negative checks. Task declarations determine which checks apply. Its authors report a duplicate-checkout trace with loyalty state 310 rather than 155, and explicitly identify a false negative in the semantic judge's contradiction flag. That is an author-reported result, not our independent replay of the original trace. Paper, sections 2.2–2.3 and Appendix D

Our internal probe supplied synthetic traces, states and assertions directly to the pinned grading and state-validation pure helpers. It ran under Python 3.12.9 without a live business environment or a model judge. The synthetic balance is an experimental field that distinguishes effects; it is not the original paper's task record. All nine cases matched their predefined expectations. The central controls were:

Synthetic trace and asserted conditionObserved verdictWhat the verdict establishes
One checkout; subsequence contract, no balance assertionPassRequired action and relative order exist
Two checkouts; same sparse contractPassThis contract does not exclude the duplicate
Two checkouts; exact named-tool orderRejectThe extra named checkout enters the comparison
Two checkouts; expected balance 155RejectActual balance 310 violates declared state

Adding the relevant assertion distinguishes these synthetic outcomes. This says nothing about the rate of duplicate transactions in live systems, or whether every benchmark task omits the same checks. The implementation identity is the pinned grading source.

Exact order still has a filter boundary ​

The examined implementation uses multiset containment for action checks. Subsequence order permits interleaving and duplicates. Exact order is stricter, but it first filters the trace to tools named in the contract. Constraining the sequence of named tools and excluding every other tool from the complete trace are different predicates.

The internal counterexample calls checkout and then an unlisted deletion tool while retaining balance 155. Exact named-tool order and the balance assertion can both pass. Explicitly forbidding the deletion action rejects the attempt. The negative check concerns the attempted call, without requiring it to be treated as a successful business effect. Order and forbidden checks

State checks have a parallel boundary. Forbidding a field from equalling “Deleted” does not require it to remain at its initial “Open” value. A synthetic “Closed” value passes that forbidden-value check; expanding expected state to require “Open” rejects it. An unchanged-state control passes the expanded assertion. These are declared-leaf semantics, not a guarantee that an entire protected record remains unchanged. State-validation source

Our interpretation is that assertion strength and coverage are distinct dimensions. Conjoining more declared predicates does not automatically constrain undeclared tools or fields.

Whole-state comparison asks a different equivalence question ​

A pinned historical tau-bench evaluator compares the final database hash against one produced by reference actions and checks required output strings. This offers a concrete alternative to sparse state predicates. It can capture state differences omitted from those predicates, but final-state equality alone does not establish the absence of prohibited intermediate attempts. Historical evaluator

Neither design is universally preferable. Subsequence semantics may intentionally permit unrelated reads and safe retries; whole-state identity may reject legitimate equivalent outcomes. The historical project warns that its tasks are outdated. We did not execute it or compare current rankings. The useful comparison concerns which business invariants acceptance should preserve and which process differences it should permit.

For governed digital employees, the supported engineering implication is to expose asserted predicates, unasserted scope, action multiplicity and side-effect boundaries alongside completion. Prohibited intermediate attempts and protected final fields need their own stated basis. This is an acceptance-evidence design recommendation, not deployment certification.

Where the evidence stops ​

Public source code is the project's own primary implementation material, not independent validation. The paper's duplicate checkout is author-reported; the table describes our internal isolated-helper experiment; separating strength from coverage is our interpretation. We have no independent third-party reproduction of this experiment. Another project's existence does not supply one.

We did not replay the full benchmark, original trace archive or model judge. The results cannot estimate live failure rates, rank systems or establish transfer across domains. Publication, citation, review and version status establish provenance and traceability rather than general validity. The generated editorial cover illustrates the proposition and is not experimental data.

How will missing predicates be found? ​

The unresolved questions are how contract review can systematically detect omitted protected fields and forbidden intermediate effects, and how safe retries can remain distinguishable from duplicate business effects. These probes show that faithful contract execution can still answer less than its reader assumes. Further evidence needs to address omission discovery itself, rather than adding another completion boolean without a coverage account.

Last updated: