How much of a published statistic belongs to the choice upstream of it: a protocol, and five applications

Abstract

A quantitative claim about an object is usually published as a single number. Between the object and that number there is always a coding decision — which signs count as one sign, which threshold counts as a detection, which weights combine the components of a score — and the decision is normally made once, by one group, and then not mentioned again. This note applies robustness analysis — the multiverse and vibration-of-effects tradition — to a case that tradition does not cover: statistics computed on a whole corpus, where there is no sampling distribution and therefore no standard error to divide by. The modelling spread is then the only spread there is, and the denominator has to be built from a null. We report the result as a ratio R: the spread the statistic shows across the coding choices competent practitioners have actually published, over the effect the statistic exists to detect, each measured against a null recomputed under its own coding. R is not a p-value and not an accusation; it is an answer to the question what is this number a property of. We give the protocol, the conditions under which it is meaningful, five worked applications — an undeciphered script, a transit-detection threshold, two published archaeoastronomical maps, and two machine-learning leaderboards — and a list of the ways the protocol misleads when it is run carelessly, each of which we have made. In two of the five the audit returns less than it set out to find, which is reported at the same length as the cases where it returns more. The protocol is a restricted case of existing robustness analysis and adds little to its statistics; section 3 states exactly which of its parts are and are not new, having first withdrawn a claim that a reading of the prior literature destroyed.