Ilpo Väätäinen

How Much of a Leaderboard Ranking Survives Its Own Sampling Error? A Paired-Resolution Audit of Five AI Benchmarks

A benchmark leaderboard is read as a stopwatch: the rank beside a model is treated as a fact. Yet every entrant is scored on the same items and publishes which it solved, so a leaderboard is a paired experiment, and its ordering can be tested rather than trusted. We test each adj...

The Benchmark Number Is Not the Deployment Number: Rank Collapse and Base-Rate Collapse, Measured Across Four Domains

A leaderboard reports one number per model. Two things that number silently discards are how many independent axes of skill it collapses into a single scalar, and what population its accuracy was measured on versus where the model is used. This paper measures both, on public data...

How much of a published statistic belongs to the choice upstream of it: a protocol, and six applications

A quantitative claim about an object is usually published as a single number. Between the object and that number there is always a coding decision (which signs count as one sign, which threshold counts as a detection, which weights combine the components of a score), and the deci...

A Pre-Registered Configuration Test for the Göbekli Tepe Pillar 43 "Date Stamp"

Two published readings assign the animal carvings of Göbekli Tepe's Pillar 43 to constellations and conclude that the pillar encodes a date connected to the Younger Dryas event. This paper audits the published probability and then runs the spatial test both sides have pointed to...

How the choice of sign inventory moves the statistics of an undeciphered script: rongorongo under three published readings, with the Indus script as an outside check

Statistical arguments about undeciphered scripts, whether a sign system "behaves like language", are computed on a transliteration, and a transliteration presupposes a catalogue of signs. For rongorongo the published catalogues disagree by an order of magnitude, from about fifty...

How Much of a Leaderboard Ranking Survives Its Own Sampling Error? A Paired-Resolution Audit of Five AI Benchmarks

A benchmark leaderboard is read as a stopwatch: the rank beside a model is treated as a fact. Yet every entrant is scored on the same items and publishes which it solved, so a leaderboard is a paired experiment, and its ordering can be tested rather than trusted. We test each adj...

How much of a published statistic belongs to the choice upstream of it: a protocol, and five applications

A quantitative claim about an object is usually published as a single number. Between the object and that number there is always a coding decision — which signs count as one sign, which threshold counts as a detection, which weights combine the components of a score — and the dec...

How the choice of sign inventory moves the statistics of an undeciphered script: rongorongo under three published readings, with the Indus script as an outside check

Statistical arguments about undeciphered scripts—whether a sign system "behaves like language"—are computed on a transliteration, and a transliteration presupposes a catalogue of signs. For rongorongo the published catalogues disagree by an order of magnitude, from about fifty ba...

How much of a published statistic belongs to the choice upstream of it: a protocol, and five applications

A quantitative claim about an object is usually published as a single number. Between the object and that number there is always a coding decision — which signs count as one sign, which threshold counts as a detection, which weights combine the components of a score — and the dec...

A Pre-Registered Configuration Test for the Göbekli Tepe Pillar 43 "Date Stamp"

Two published readings assign the animal carvings of Göbekli Tepe's Pillar 43 to constellations and conclude that the pillar encodes a date connected to the Younger Dryas event. This paper audits the published probability and then runs the spatial test both sides have called for...

The Catalogue Is the Instrument: how the choice of sign inventory moves the statistics of an undeciphered script

Statistical arguments about undeciphered scripts — whether a sign system "behaves like language" — are computed on a transliteration, and a transliteration presupposes a catalogue of signs. For rongorongo the published catalogues disagree by an order of magnitude, from about fift...

The Catalogue Is the Instrument: how the choice of sign inventory moves the statistics of an undeciphered script

Statistical arguments about undeciphered scripts — whether a sign system "behaves like language" — are computed on a transliteration, and a transliteration presupposes a catalogue of signs. For rongorongo the published catalogues disagree by an order of magnitude (about 50 to abo...

The Catalogue Is the Instrument: how the choice of sign inventory moves the statistics of an undeciphered script

Statistical arguments about undeciphered scripts — whether a sign system "behaves like language" — are computed on a transliteration, and a transliteration presupposes a catalogue of signs. For rongorongo the published catalogues disagree by an order of magnitude (about 50 to abo...

The Catalogue Is the Instrument: how the choice of sign inventory moves the statistics of an undeciphered script

Statistical arguments about undeciphered scripts — whether a sign system "behaves like language" — are computed on a transliteration, and a transliteration presupposes a catalogue of signs. For rongorongo the published catalogues disagree by an order of magnitude (about 50 to abo...

The Catalogue Is the Instrument: how the choice of sign inventory moves the statistics of an undeciphered script

Statistical arguments about undeciphered scripts — whether a sign system "behaves like language" — are computed on a transliteration, and a transliteration presupposes a catalogue of signs. For rongorongo the published catalogues disagree by an order of magnitude (about 50 to abo...