Hidden or undetected? Quantifying the boundary between structural bias and statistical noise in peer review

Abstract

Purpose Peer review shapes not only the evaluation of scientific claims but also whose work gains visibility and influence. A growing body of evidence shows that review outcomes can mirror structural inequities tied to authors' identities, language backgrounds, institutional positions, and geographic locations. Most of the existing findings come from observational submission data, which can document disparities without revealing which forms of bias are actually detectable under the noise and constraints of real review systems. This raises an equity concern: some inequities may persist not because they are small but because standard analytical approaches lack the statistical power to uncover them. Design/methodology/approach This study introduces a stylized simulation of the peer-review process that includes field status, topical fit, multidimensional quality signals, and socially patterned bias attributes such as institutional prestige, Global South affiliation, past co-authorship, and language-related disadvantage. The framework is paired with Monte Carlo power analysis to examine which structural disparities can be empirically recovered under realistic levels of reviewer disagreement, correlated signals, and assignment variability. Findings The results show that reviewer noise and structural correlation sharply constrain detectability. We found that with sample sizes around 200, we recover only moderate-to-large effects consistently, while smaller or more relational disadvantages remain difficult to distinguish from noise. Consequently, the absence of significant disparities in observational audits cannot be interpreted as evidence of equitable review. Originality/value By delineating the boundary between detectable and latent inequities, the study offers practical guidance for designing more informative fairness audits and more accountable peer-review systems.