Skip to content
Experiment Report
The Same Workflow Results Can Produce Different Rankings
Research

The Same Workflow Results Can Produce Different Rankings

Across 25 released configurations, partial score and complete acceptance reverse 34 pairwise orderings robustly to rounding. Domain weighting reverses six more, only three beyond rounding uncertainty. State the success objective before interpreting the ranking.

2026-11-11中文 →

The Same Workflow Results Can Produce Different Rankings ​

Across 25 released configurations, partial score and complete acceptance reverse 34 pairwise orderings robustly to rounding. Domain weighting reverses six more, only three beyond rounding uncertainty. State the success objective before interpreting the ranking.

Higher scores need not mean more complete deliveries ​

Two configurations reported by WorkflowArena expose the distinction directly. The high-reasoning setting of Claude Opus 5.5 has a continuous score of 80.21 and complete acceptance of 34.11%; the maximum-reasoning setting scores slightly lower, at 79.94, but completes acceptance more often, at 36.95%. One leads on partial quality, the other on complete acceptance. These are the authors' released aggregates, not model measurements rerun here.

Continuous scoring preserves information about partially completed work. Complete acceptance asks whether all relevant task conditions were satisfied. The paper's software-engineering domain even pairs a score of 96.98 with complete acceptance of 17.65%. The two can coexist: a delivery may satisfy most scored items while missing an indispensable condition. Where procurement or deployment requires complete delivery, the continuous score is not a success probability.

The benchmark contains 145 tasks across eight domains and 17 task forms. Hidden verifiers assess artifacts and applicable process constraints; worker count or topology is not directly rewarded. This article examines interpretation of the same aggregates, rather than inferring that one organization must be better from its leaderboard position.

Compare the orderings pair by pair ​

The same-run internal statistical reanalysis uses only the pinned release already covered by reading: the official repository commit beginning 637adec and paper version 1. It does not merge the later paper version described on the project page. Twenty-five configurations yield 300 unordered pairs. For each pair, opposite nonzero signs for the continuous-score difference and complete-acceptance difference define a strict reversal.

The completed study records 34 reversals. Analysis then treats every displayed value as rounded to two decimal places, with uncertainty of at most ±0.005. All 34 retain their difference signs under those bounds in both metrics. Metric choice therefore changes the ordering in this table beyond display-rounding ambiguity. The 300 pairs share configurations: 34 reversals are not 34 independent experiments or a statistical-significance result.

Comparing a metric with itself yields zero reversals as a direction-check control. Reconstructed equal-domain means differ from the released aggregates by at most about 0.005, consistent with displayed rounding. These controls support the aggregation interpretation. They neither validate hidden graders nor turn author results into an independent replication.

Equal domains and equal tasks answer different questions ​

Released overall acceptance uses an equal-domain mean. The domain counts are 17 software-engineering, 18 operations, 16 science, 24 mathematics, 16 finance, 22 law, 14 content-delivery and 18 3D tasks, totaling 145. Equal-domain averaging gives the 14-task and 24-task domains the same total weight. Task-count weighting gives every task the same nominal weight. Either can be defensible, but they express different objectives.

The existing internal reanalysis weights the same domain acceptance rates by task counts and finds six strict reversals. Only three remain robust under the rounding-bound check; the other three cannot be described as reliable changes. Task-weighted rates here are approximate reconstructions from rounded domain percentages, not recounts of task-level records.

Consequently, the best released 36.95% does not mean that 36.95% of all 145 pooled tasks passed. For evaluation engineers, declaring domain importance before ranking configurations is more reviewable than selecting a favorable aggregation after seeing results. We recommend retaining domain-level results alongside the preselected summary. That evaluation-design interpretation does not make task weighting a replacement for the paper's equal-domain policy.

Transfer gains do not isolate topology ​

The same research compares fixed execution models receiving workflows from different source models. The authors report mean continuous-score gains of +6.4125 and +6.2675 across four executors for Opus-authored static and dynamic settings relative to each executor's self-dynamic setting. In the existing internal leave-one-executor-out checks, both means stay positive after each omission. The descriptive sign is therefore not driven solely by one executor, but these checks are not repeated trials or confidence intervals.

Those limits do not negate the potential value of transfer. They constrain attribution: workflows carry organization and instruction knowledge written by the source model. Static programs are frozen before execution, whereas dynamic authors can observe execution and revise. Without separating these factors, score changes cannot be assigned solely to graph structure. Continuous-score gains are also not percentage-point gains in complete acceptance.

The transfer study uses a purposively harder 32-task subset and only one execution per task. Repeated runs, held-out tasks and controls separating instruction knowledge from organization are needed to assess stability. None was executed in this production task.

Evidence and sources: the reanalysis boundary ​

Paper version 1 specifies tasks and evaluation protocol. The official pinned repository supplies task data, evaluators and aggregates while explicitly excluding execution frameworks and per-task experimental traces. Both are author primary material; agreement is not outside independent evidence. Internal evidence here means the preceding research stage's completed aggregate reanalysis and rounding checks. Recommendations are our interpretation.

One execution per task, missing variance estimates for close rankings, and missing human-agreement measurements for content and 3D semantic judges constrain capability judgments. Thirty-four robust reversals support the claim that metrics change this table's order, not that a model is truly superior. Three robust weighting reversals establish sensitivity to aggregation policy, not a universally correct weight scheme.

Publication and citation locate sources without establishing validation or endorsement. Peer review is not proof; self-reported results are not independent evidence; implementation success does not establish general validity. Readers can now check whether their target is partial quality or complete acceptance, freeze domain weights, and demand repeated measurements for close rankings. A single leaderboard place cannot supply certainty missing from the data.

Internal research material: pairwise reversals, domain weights and transfer sensitivity. Rounding-bound checks. These files preserve completed internal calculations, not an independent replication.

Last updated: