Conditional Independence Fails on MultiPref: Auditing the Assumption Behind Annotator Reliability Scores

Abstract

Reliability-weighted RLHF pipelines fit a Dawid–Skene model to multi-annotator preferencedata, score each annotator by the mean diagonal of their confusion matrix, and downweight or discard the low scorers. That summary statistic cannot distinguish a careless annotator from one with a systematically different preference, and the distinction is only meaningful if annotator deviations are conditionally independent — the model’s own core assumption. We test it directly. Reproducing RGPO’s estimation stage on MultiPref exactly (tie rate 12.27% →29.03%; mean reliability 0.6459 against a published ≈0.646), we run a parametric bootstrap in which the null is conditional independence. It is rejected decisively: four statistics reach the smallest attainable p-value at B=1000, with |z|up to 8.6. The reliability score, and the 2,237 items (21.4% of the corpus) its pipeline moves into the discarded bucket, are therefore computed under a model the data rejects. Neither of two confound models — a smooth item-difficulty drop and a mixture of item-difficulty classes — reproduces the observed signature. Using MultiPref’s labelled crowd/expert partition we find no group-aligned dissent, an informative null with 70–100% power against such a faction at β ≥0.10. What remains — whether the excess structure is a preference faction cutting across that partition — is not settled by this dataset, whose 2.45 items of overlap per annotator pair leave the direct test unpowered. We report three replication traps that make this analysis easy to get wrong, and release all code