Ephemeral Verifiable Benchmarking for AI Systems

Abstract

(Abstract, 1:1):Static benchmarks suffer from contamination, gaming, and saturation: their fixed items become known or reused. We propose Ephemeral Verifiable Benchmarking (EVB): each test book is generated fresh on demand from abstract mathematical schemas, independently verified, and issued to one requester as a unique artifact, so its items could not have been seen in advance. Integrity thus rests on freshness, verifiability, and per-item uniqueness rather than secrecy, removing exact contamination by construction and reducing structural contamination. A prototype with 52 schemas, five exact answer formats, independent verification, and a calibrated base-2 difficulty scale with an IRT ability estimate demonstrates feasibility.