Benchmarking Post-Training Under Constrained Compute: Protocols for Question Generation, Variance-Controlled Evaluation, and Judge Auditing

Abstract

Preprint of the methods paper. Companion findings paper: "What Determines the Direction of Post-Training Gains? A Preregistered Multi-Scale Study of Method, Scale, and Base-Model Identity" (Zenodo, DOI: 10.5281/zenodo.23409394). LLM-as-a-judge scoring is increasingly the default way to measure post-training gains on reference-free generation tasks. Yet the measuring instrument itself is rarely audited in the studies it serves: scores drift when the scoring service changes between evaluation epochs, and generation stochasticity makes each reported score a single draw from an unmeasured distribution. We present a protocol that treats evaluation as an experimental object in its own right. The pipeline controls the question distribution by design, audits candidates dual-blind with third-model arbitration, and calibrates between-generation variance per model scale through an $n=3$ regeneration protocol with promotion criteria frozen in writing before any scored comparison. A drift detection procedure re-judges fixed answers across judge epochs and counters drift by regeneration rather than post hoc correction. Applied to a multi-scale post-training study, the protocol detected a systematic judge shift between epochs and promoted only one of four preregistered comparisons. Re-applied to a medical-domain study on the same judging stack, the frozen criterion decided all nine trained-versus-base comparisons, four of them losses. We demonstrate that aggregate ordering behavior is consistent with the expected ordering on a graded quality ladder in the dimensions where the ladder discriminates, with both deviations noted; item-level failures are reported as measured limitations rather than assumed-away reliability.