How much of a published statistic belongs to the choice upstream of it: a protocol, and six applications

Abstract

A quantitative claim about an object is usually published as a single number. Between the object and that number there is always a coding decision (which signs count as one sign, which threshold counts as a detection, which weights combine the components of a score), and the decision is normally made once, by one group, and then not mentioned again. This note applies robustness analysis, the multiverse and vibration-of-effects tradition, to a case that tradition does not cover: statistics computed on a whole corpus, where there is no sampling distribution and therefore no standard error to divide by. The modelling spread is then the only spread there is, and the denominator has to be built from a null. We report the result as a ratio R: the spread the statistic shows across the coding choices competent practitioners have actually published, over the effect the statistic exists to detect, each measured against a null recomputed under its own coding. R is not a p-value and not an accusation; it is an answer to the question what is this number a property of. We give the protocol, the conditions under which it is meaningful, six worked applications (an undeciphered script, a transit-detection threshold, two published archaeoastronomical maps, two machine-learning leaderboards, and ten clinical variant-pathogenicity predictors) and a list of the ways the protocol misleads when it is run carelessly, each of which we have made. In two of the six the audit returns less than it set out to find, which is reported at the same length as the cases where it returns more. The protocol is a restricted case of existing robustness analysis and adds little to its statistics; section 3 states exactly which of its parts are and are not new, having first withdrawn a claim that a reading of the prior literature destroyed. Version 4 (2 September 2026) changes no result, table, figure or number. The prose is repunctuated: 29 em dashes are unpacked into commas, colons, parentheses and separate sentences, in the paper and in this description. The evidence table is unchanged. Version 3 (30 August 2026) corrected the title, which said five applications where the abstract said six, and replaced a PDF that predated the prior-work correction: Boufourou (2026), Sanjeev (2026) and Sapone (2026) appeared while the note was being written, and two of its claims are narrower because of them. Version 5 (18 September 2026). No measurement changed: the four ratios in the case table, the permutation and configuration p-values and the reprojected precisions are all as computed. Nine statements about other people's work did not survive being read against the source. Two change what a case is evidence for. The two published readings of one carved pillar were described as supporting the same historical conclusion; they do not, and the second author later published a critical evaluation of the first paper, so the sentence turned a critic and his subject into allies. The rongorongo set was described as three whole editions by different scholars; it is three published granularities of one transliteration, which the source paper for that case states in its own data section. The remaining seven: the 131,072 regressions are Munoz and Young (2018), not Young and Holsteen (2017), the root of which was a file in the working directory named for the second paper and containing the first; Sapone reports that the calibration error leaves the recovered parameters plausible, not unchanged, which is the whole blind spot; the deflation comparison is against the expected-maximum formula as commonly applied rather than a published correction of that leaderboard, which does not exist, and Bailey and Lopez de Prado devote an appendix to the dependent case; Sanjeev compares three published methods with a fourth of his own rather than the estimators of rival groups; the normalised conditional entropy is introduced in Rao et al. (2009), and there is no "Rao-Sproat scale"; the clinical benchmark has 2,525 proteins, of which 500 are this author's sample; and Steegen et al. ran two studies. Two further items concern the paper's own process: one case is absent from the table because its inventor is someone the author knows personally and has not asked, not because it could not be verified, and the opening example is this author's own computed value rather than a quotation.