TreeBench-Audit: Failure-Driven Validation of Machine-Generated Hierarchical QA Benchmarks

Abstract

TreeBench-Audit is a failure-driven audit of a machine-generated hierarchical question-answering benchmark built from U.S. federal regulatory text. The study treats benchmark construction as an executable pipeline in which source parsing, question generation, answer construction, evidence alignment, and label verification can fail independently while still producing plausible-looking artifacts. The audit identified two major defects in earlier TreeBench releases: question–evidence leakage affecting 73.2% of the deprecated 861-question artifact, and identifier collisions that silently overwrote 326,817 parsed nodes. All retrieval results computed on those artifacts are withdrawn. The corpus was rebuilt using collision-free path-derived identifiers, producing 988,437 nodes, and an 11-gate validator was tested against 17 injected corruption types. The paper also evaluates exact structured-label reproducibility on 113 repaired candidates using three blinded model families. The models agreed on coarse disposition for 102/113 items (90.3%), but only 4/113 items (3.5%) reached exact agreement on claim type, normalized value, support-node set, and evidence spans. This result measures label reproducibility under a strict promotion protocol; it is not an accuracy estimate, and model consensus is not treated as human-validated ground truth. This release provides: the TreeBench-Audit preprint and LaTeX source; parser and validation code; the 11-gate validation framework; mutation-based regression tests; frozen manifests and aggregate verification reports; the semantic correction withdrawing the earlier negative-control interpretation; documentation of the versioned audit and withdrawal process. This record does not contain the full rebuilt corpus or a human-gold benchmark dataset. The repaired records remain candidates for future human validation and must not currently be used as authoritative benchmark ground truth, a leaderboard, or legal advice. Earlier TreeBench versions and their retrieval claims are retained only for historical and forensic reproducibility. Author: Sahil SoniLicense: CC BY 4.0 for the paper, documentation, schemas, manifests, and reports; MIT License for included source code.