Shiva Koreddi

Conditional Independence Fails on MultiPref: Auditing the Assumption Behind Annotator Reliability Scores

Reliability-weighted RLHF pipelines fit a Dawid–Skene model to multi-annotator preferencedata, score each annotator by the mean diagonal of their confusion matrix, and downweight or discard the low scorers. That summary statistic cannot distinguish a careless annotator from one w...