Adjusted Positive Agreement: A Chance-Corrected Agreement Measure for Unconstrained Pile Sorts

Abstract

Unconstrained pile sorting, also known as free pile sorting or card sorting, is an important method for studying how people organize items within a domain. Yet commonly used measures for estimating respondent agreement struggle with the lumper–splitter problem. As a result, apparent similarity between respondents may reflect propensities to lump or split rather than shared conceptual agreement. This article introduces the adjusted positive agreement (APA), a pair-counting coefficient designed to address this problem. APA compares the item pairs that two respondents place together, adjusts this overlap for the total number of co-placed pairs produced by each respondent, and scales the result relative to the smaller of the two respondents’ co-placement totals. Across two simulation conditions, APA preserved agreement with less influence from lumping and splitting than Pearson correlation and the adjusted Rand index. The findings support APA as a basis for analyzing how cultural models are distributed across respondents.