The Benchmark Number Is Not the Deployment Number: Rank Collapse and Base-Rate Collapse, Measured Across Four Domains

Abstract

A leaderboard reports one number per model. Two things that number silently discards are how many independent axes of skill it collapses into a single scalar, and what population its accuracy was measured on versus where the model is used. This paper measures both, on public data, with every figure independently re-verified through a second code path before inclusion. Rank collapse: on ProteinGym, the standard variant-effect-prediction benchmark, a null-gated decomposition of the 217x97 assay x model matrix shows the single ranking hides at least three specialization axes (singular values 7.87/6.60/5.88 against a permutation null of 4.46/4.32/4.22). The sharpest axis, structure-vs-sequence models on protein stability, is powered across all 64 Tsuboyama stability assays (64/64, median gap +0.346) and traced to a chemistry x exposure mechanism: structure-aware models win on surface aromatic substitutions (p=8e-8) and lose on surface charge substitutions, converging in the buried core, after two competing hypotheses (buried-core packing; general hydrophobicity) were tested and killed. The same null-gated test applied to three unrelated benchmarks (MTEB embeddings, SWE-bench code-agent evaluation, LMArena chat preference) finds a hidden axis in every one that resolves at all. Base-rate collapse: ten clinical variant-pathogenicity predictors report AUCs of 0.81-0.98 on a benchmark that is 65% pathogenic; reprojected by Bayes to the 0.1-1% prevalence a real screening population presents, the best predictor's flag precision falls from 93% to 12% (1.3% at 0.1%). The identical arithmetic, applied to AI-generated-text detection on GPT-4 output, takes the strongest detector's benchmark precision of 79% to 16% at a 5% real-world machine-text rate -- about five false accusations per genuine catch -- and shows the classifier lineage behind many commercial "AI checkers" scores AUC=0.545 on GPT-4, indistinguishable from a coin flip. Genomics and generated text share the identical failure shape because the underlying arithmetic is the same theorem; only the base rate changes. The paper releases two reusable, self-tested primitives -- a rank-dimensionality auditor and a Bayes recalibrator -- and the full analysis pipeline. Version 2 (18 September 2026). The Bayes reprojection this paper is about is unchanged, and so are the dimensionality results, the frontier analysis and every reported AUC. What changed are eighteen statements about other people's work and about this paper's own inputs, each read against its source. The most serious: Singh et al. were credited, "independently", with Huang, Shen, Wei and Broderick's result that dropping a very small share of preference data can change a preference arena's top model; Grimm et al. (2015) were cited for the dependence of predictive value on prevalence, which their paper never discusses, and their type-1 circularity is restated correctly as overlap between training and evaluation variants in a study that does not involve ClinVar; and a paraphrase-stability citation named two of its four authors (Lunardi, Della Mea, Mizzaro and Roitero). Numbers corrected: the clinical figures come from a 500-protein sample of a 2,525-protein benchmark; the base-rate collapse is 7.7-30.4x at 1% prevalence and 69.5-304.3x at 0.1%, not "6-70x"; precision under 5% at 90% recall applies to four predictors of ten, not half; the Tsuboyama assays carry 117,811 variants, not ~40,000; MTEB's 19.7x range is of standard deviation and the board is English v1; and the SWE-bench matrix holds the 468 of 500 instances some submission solved. The claim that many commercial "AI checkers" descend from OpenAI's RoBERTa detector had no source and is withdrawn. Finally, version 1 said the two reusable primitives shipped with the release while the release held only the PDF; they are deposited here. All of it is listed in "Corrections in this version".