Finite-sample decision risk in unseeded quantum compiler benchmarking

Abstract

Draft preprint. Not submitted to a venue. Quantum compiler benchmarks are used to accept or reject changes to production transpilers, but the compilers they measure are stochastic. At the pinned revision of IBM's Benchpress benchmark suite, its Qiskit gym compiles without passing seed_transpiler, so every gate count it reports is one draw from an unmeasured distribution — while the BQSKit gym in the same repository does seed its compiler. We quantify the consequence as finite-sample decision risk: the probability that a regression verdict computed from k runs per version disagrees with the verdict implied by the long-run mean of the same measurement. On bv_n140 mapped to a heavy-hex lattice, measured over 400 seeds per version across 21 OS processes, the change from Qiskit 1.4.3 to 2.0.0 has a long-run mean of +5.37% (95% CI +4.27% to +6.50%). The suite's own protocol — three unseeded runs per version — reports it as a ≥+10% regression 24.4% of the time (95% CI 19.5–31.1%). A disjoint 200-seed sample across ten fresh processes reproduces this at 22.6%. Twenty runs per version, roughly 40 hours of compute, still leaves 3.7%. A pre-registered replication across 39 circuits is reported in full: the analysis code was committed before the data existed, and the commit ordering is included in this archive as COMMIT_TIMESTAMPS.txt. The pre-registered primary endpoint is 12 of 26 eligible circuits (46.2%, Wilson 95% CI 28.8–64.5%), reported alongside a cluster-robust interval of [17.4%, 81.0%] that reflects algorithm-family clustering. What this archive contains: the paper, the complete analysis and measurement code, and all 41,790 raw per-seed measurements. verify.py re-runs six checks — the toolchain pin, the test suite, the numeric inventory, the replication artifact, a proof of a withdrawn analytical claim, and a recomputation of every quantitative claim in the manuscript against the text — in about two minutes. Four claims from earlier phases of this work were withdrawn on the record after adversarial review, and the withdrawals remain in the repository with the evidence that defeated them. Passing seed_transpiler removes false positives on the circuits tested but was worse on three of four circuits exhibiting false negatives, and is not presented as a general remedy. Changes in v2 (2026-09-10). This version corrects two overstatements carried by the v1 manuscript and adds one new pre-registered result. No measurement changed, and no reported figure moved. Withdrawn. v1 stated that Benchpress sets no transpiler seed anywhere. That is narrowed to the claim the evidence supports: at the pinned revision, the Qiskit gym compiles without passing seed_transpiler — while the sibling BQSKit gym does seed its compiler, and six further gyms were not assessed. v1 also stated that the work identifies the mechanism that governs the effect. The paper now reports what predicts the risk and explicitly declines to call that association a compiler mechanism. Added: a pre-registered cross-machine determinism control. All 216 per-seed gate counts are identical across two physical machines with different CPU vendors (AMD64 AuthenticAMD, 16 cores, Python 3.10.10; Intel64 GenuineIntel, 8 cores, Python 3.10.21), over all three Qiskit versions including the 1.4.3 → 2.0.0 pair the primary study uses. The design was committed before the second machine existed and inherits the frozen six-circuit, twelve-seed selection, so it introduces no selection freedom. Stated scope limit: both machines ran rustworkx 0.18.1, so this cannot separate “the CPU does not matter” from “the graph library was identical, so the CPU never had the opportunity to matter”. The primary 200-seed study itself ran on one machine. Also in this archive. The settled-question ledger now holds 33 entries, each naming the attack, who made it, the command that settled it and the result; it includes three rounds of external adversarial audit of the cross-machine comparator itself, in which fourteen defects were found in the checking instrument and none in the measurements. The cross-machine evidence, its pre-registration, the comparator and its property-based fuzz tests are included. The numeric inventory now records a provenance tier per figure (29 re-derived from raw per-seed files, 12 read from a summary table, 5 not applicable). verify.py now runs six checks rather than five, the sixth recomputing every quantitative claim in the manuscript and comparing it to the text. Two reproducibility defects in the README were fixed: a shallow clone that could not reach the pinned Benchpress revision, and a missing environment without which the primary 1.4.3 → 2.0.0 study could not be reproduced at all.