Skip to content
Experiment Report
Trace-supported and factually wrong: a probe of audit-function semantics
Research

Trace-supported and factually wrong: a probe of audit-function semantics

A pinned-function probe produces support and contradiction rates of one for the same synthetic claim. Provenance, factual verification and policy disposition need to retain their separate meanings.

2026-11-08中文 →

Trace-supported and factually wrong: a probe of audit-function semantics ​

What happens when one claim is labeled trace-supported but factually contradicted? In our internal probe, the pinned functions returned an evidence support rate of one and a factual contradiction rate of one, while still classifying the claim as grounded. This was not a financial verification exercise: the inputs were hand-authored verdict labels, and the experiment checked how code interprets them.

The reader problem is concrete. An agent can faithfully cite an incorrect tool result. Its answer has a traceable origin without becoming correct. Completion auditing for tool-using agents therefore needs to retain provenance, external verification and policy disposition separately. An aggregate percentage cannot simply be renamed factual accuracy.

One claim, two objects of judgment ​

The AgenticFinBench paper separates provenance from factual verification. Provenance asks whether tool outputs or calculations in the execution trace support a claim. Verification asks whether it agrees with checkable facts. A grounded claim can still be verified, refuted or unverified. This is the paper's public method definition, rather than our independent validation of its judge's ability. Paper, Section 4.2

The authors' cases preserve that distinction. One inherits a numerical error from retrieved notes: it remains grounded while retaining an external refutation. Another remains unverified because the specified range cannot be established or cleanly refuted. These are source-reported cases, rather than independent reproductions. Paper, Appendices G and H

Traceability is useful, but it answers a provenance question. Treating it as correctness can hide a tool's own error behind faithful citation.

Nine verdict combinations, two functions ​

The same-day Analysis stage used Python 3.12.9 to extract two unchanged function bodies through the abstract syntax tree: one computes statistics and the other classifies outcomes. It enumerated three trace verdicts against three factual verdicts, plus an empty-input control. It did not call a model, semantic judge, financial dataset or complete benchmark. The source is pinned to commit b3a575b76aad765ea1f46759c366897b5312311a; that pin binds the inspected public code, not the manuscript experiments. Statistics source, outcome source

Two singleton inputs make the boundary especially clear. Each row is internal experimental evidence. A value of one means the sole claim in that synthetic input enters the corresponding set; it is not 100% accuracy on real financial tasks.

Trace verdictFactual verdictOutcomeEvidence support rateFactual contradiction ratePolicy pass rate
SupportedContradictedGrounded111
UnsupportedUnresolvedUnverified001

Source: complete internal synthetic-label results. For the empty control, claim count is zero and all three rates are null. They must not be displayed as zero errors or universal success.

The first row combines two different counting rules. Evidence support counts a claim when either the trace verdict or the factual verdict is supported; factual contradiction independently retains a factual refutation. The outcome function preserves trace-supported grounding. Membership in both sets is therefore compatible with the definitions, rather than an arithmetic contradiction.

The second row identifies another interpretation trap. Default policy can pass an unsupported, unresolved claim while the outcome remains unverified. Policy passage describes disposition, not newly acquired factual evidence. The corresponding source identifiers are:

text
evidence_support_rate
factual_contradiction_rate
policy_pass_rate

Why this does not contradict the paper ​

If the intended question distinguishes origin from correctness, retaining both judgments is necessary. High support alongside factual contradiction does not establish benchmark invalidity. It establishes the reader's need for metric definitions. Our interpretation is that the problem arises when several questions are collapsed into a single completion-quality label, rather than in the multi-axis record itself.

The public repository also distinguishes public-runner diagnostics from paper outcomes and says the full manuscript responses, traces and claim judgments are absent. Paper release wording and the repository's available material need separate treatment. Executable code alone does not establish paper reproduction. We neither recomputed manuscript scores nor tested judge reliability. Repository documentation

What a completion record should retain ​

For evaluation engineers, the probe supports a bounded design recommendation: retain the provenance verdict, factual verdict and unresolved state for each claim; expose numerators, denominators and default policy in aggregates. A broad pass badge should not overwrite an already recorded factual refutation.

A singleton record might show “source-supported: 1/1,” “factually contradicted: 1/1” and “policy-passed: 1/1” together. This presentation does not improve correctness. It lets a reader distinguish faithful use of an incorrect tool result from an unsupported fabrication. Once individual verdicts are discarded in favor of a final percentage, that difference cannot be reconstructed from the aggregate. This is our engineering interpretation of function semantics, rather than a measured improvement in a deployment comparison.

Evidence and sources ​

We distinguish public-code observations, author reports and internal experiments. Code shows what particular functions do; paper cases show how the authors use the concepts; synthetic inputs show how known labels pass through the functions. None supplies third-party validation of the factual judge. We have no independent evidence here for error incidence or model rankings.

Direct sources are the authors' AgenticFinBench Section 4.2 and appendices (preprint v1), the author-maintained pinned audit source and scoring source (code), and the internal experiment record (2026-11-08, environment, inputs, outputs and source hashes). Repository documentation bounds reproducibility; it does not replace result evidence. The cover is a generated editorial interpretation of the article, rather than an experimental apparatus or result image.

The next layer remains untested ​

This probe did not verify financial facts, obtain expert relabeling, execute the complete benchmark or evaluate a semantic judge. Hand-authored verdicts exclude the judge's errors and cannot estimate how frequently real tasks are grounded but wrong.

To advance to judge quality would require original responses, complete traces, external evidence aligned in time and scope, and genuinely independent relabeling. Reproducing manuscript scores would additionally require frozen manuscript inputs and judgments. The present result establishes the semantic boundary of two functions. Whether real factual verdicts remain stable under identical evidence is a separate question.

Last updated: