The WEIRD instrument problem and systematic bias

Abstract

Psychological data from Western, Educated, Industrialized, Rich, and Democratic (WEIRD) populations seem to be more structurally complex than data from small-scale and non-Western populations. This has been hypothesized to be due to WEIRD populations having greater psychological or socioeconomic complexity. I argue that the apparent greater complexity of WEIRD psychology can instead be explained by a \say{WEIRD instrument bias} stemming from optimizing psychological instruments for WEIRD populations and adapting them to culturally dissimilar groups. Using simulation models, I demonstrate how instruments optimized for reference populations will systematically underestimate the complexity of novel populations, whether complexity stems from latent constructs or from socioeconomic niches. The models additionally predict that populations' measured complexity will negatively correlate with their cultural distance from the reference population. Consistent with this prediction, I find that a country's personality structure complexity, measured by an instrument optimized for the United States, has the strongest negative relationship with cultural distance from the United States out of 67 countries. The logic of the WEIRD instrument problem also applies to instruments optimized for a single gender, cohort, or species. Addressing it will require cautious interpretation of cross-cultural research, designing cross-cultural instruments from the ground up, and, ultimately, building better theories of causation and measurement.