Completion, legality and safety: why three metrics cannot stand in for one another
A task can score 1 on goal attainment and still fail acceptance. That need not be a statistical mistake: completing requested goals and preserving existing content can be separate checks. In another evaluation module, a true legal-plan flag can coexist with a safety violation.
These differences appeared in internal experiments that supplied synthetic reports to pinned public metric modules. For engineers interpreting multi-agent evaluations, the useful question is what a field actually checks—and which questions it leaves unanswered. Our bounded judgment is that goal completion, acceptance, trajectory legality and safety diagnostics need separate scopes; one field cannot endorse the others.
Three evidence roles matter here. Paper experiments remain author-reported. Public code establishes the computation at a particular revision. Our executed module outputs are internal experimental evidence; no independent actor has reproduced these claims. The tables below contain synthetic-input module outputs, not model success rates or production failure estimates.
Full goal progress can still fail acceptance
The MAC-Bench paper distinguishes goal-group progress from resolved rate. Resolution additionally requires applicable preservation conditions; a normal episode ending is also separate from resolution. Section 4.1 and Appendix G.2 define the authors' metrics, rather than a universal acceptance standard for business tasks.
The same-date research executed paired inputs against the pinned metric implementation, producing these internal results.
| Synthetic input | Goal completion | Progress score | Task success | Resolved-rate contribution |
|---|---|---|---|---|
| Accepted, all goals complete | 1 | 1 | true | 1 |
| Rejected, all goals complete | 1 | 1 | false | 0 |
| Evaluation error, input still carries goal value 1 | null | null | null | null |
Source: same-date internal accepted, full-progress/rejected and unavailable-evaluation controls; every input is synthetic.
The decisive boundary is at the input. The experiment supplies an acceptance verdict; it does not run a real task grader. It shows that the adapter permits full goal progress alongside failure, rather than discovering a naturally occurring task exploit.
The paper separately reports a task-level example: five goals remain complete after a template-handling mutation corrupts retained error metadata, causing preservation failure. That corruption-and-repair case remains the authors' evidence in Section 4.4 and Appendices D.3–D.4; we did not independently execute its grader. It motivates the preservation requirement, while the internal module control shows how the distinction survives in result fields. Together they still do not constitute an end-to-end reproduction.
A legal-plan label can exceed its checked scope
The pinned CoCoBench metric code computes final success, action legality, coordination-construct scores and safety violations separately. Its legal-plan flag primarily checks undeclared skills and precondition violations. It does not fold all safety diagnostics into a single Boolean.
An internal control declares two agents, authorizes one to use BreakObject, and declares its target fragile. The supplied report retains final success and gives each agent one action. The module returns a true legal-plan flag, a D1 load-allocation score of 1, and one safety violation. The allocation score describes action distribution; it is not a safety score.
Paired controls sharpen that scope. An undeclared skill makes legal-plan false. A failed interaction classified as “already clean” does not increment dependency violations, while a non-exempt failed pickup does. The latter is an output of a report classifier, rather than confirmation of the physical cause of failure in a scene.
This coexistence is not a defect verdict against the upstream implementation. Final success is supplied by the fixture, not produced by the simulator or a task grader. We have not established that every synthetic combination is naturally reachable. The narrower conclusion is that readers cannot expand “legal plan” into comprehensive safety or authorization certification, or interpret high load allocation as overall task quality.
Missingness and low signal belong beside the mean
The same internal controls examine construct engagement thresholds.
| Synthetic trajectory | Legal plan | Construct score | Sample state |
|---|---|---|---|
| Zero actions | null | null | No actions |
| One action | true | null | Low signal |
| One action from each of two agents | true | 1 | Two engagement events, scored |
Source: same-date internal zero-action, one-action and balanced two-agent controls.
Zero actions supply neither a legality counterexample nor evidence of passing legality. One action does not support this construct score either. The engagement rules in the construct and scoring definitions therefore need to accompany the score.
The acceptance denominator needs similar visibility. For one accepted, one rejected and one unavailable evaluation, the internal experiment reports two scored episodes, resolved rate 0.5 and mean progress 1, plus one explicitly excluded unavailable evaluation. That 0.5 is not an overall delivery probability across all three attempts. Exclusion is not inherently wrong; concealing the excluded population misleads readers about the denominator.
Our engineering interpretation is to retain goal attainment, acceptance, measured trajectory constraints and evidence coverage together. A short label needs its actual predicate, applicable population and missingness rule to support a decision. The two benchmarks differ in environment, tasks and denominators. Their comparison cannot yield a pooled success rate or a universal ranking of multi-agent architectures.
Evidence and sources
This article consumes completed same-date research and internal experiments; no new experiments were performed during writing. The pinned MAC-Bench revision is 1024991685762c54977ac45e1df4b6f6641a3e1b, and the CoCoBench revision is baa4dd5d9669251089c8d72ba7bb0532323b06a4. Five MAC controls and seven CoCo controls passed. Public module paths are linked above; the internal inputs, expectations and actual outputs are retained with the candidate materials.
The MAC-Bench paper supplies author-reported goal, acceptance and preservation evidence. The CoCoBench paper supplies the authors' construct, legality and success definitions. Their current public code is a locatable implementation fact, rather than third-party evidence examining the same performance claim.
Publication is not validation; citation is not endorsement; peer review is not proof. Self-reported evidence is not independent evidence, and successful module execution does not establish general validity.
Which combinations arise during real execution remains open
We did not download and execute complete task packages, original run archives or private graders, and did not run the embodied simulator. The current public commits have not been established as line-for-line equivalents of the original experimental revisions. These results cannot measure natural occurrence frequencies, reproduce paper success rates or provide safety certification.
The next research question is which report combinations actually arise from execution trajectories at fixed task and code versions, and how separate diagnostics change real acceptance decisions. Until that evidence exists, engineers can preserve and explain each check's scope. A comprehensive-sounding metric name cannot substitute for an unperformed check.