A taxonomy of reliability indices for cluster‐level constructs measured by multiple individual‐level indicators

Abstract

Abstract Generalizability theory (GT) provides a variety of intraclass correlation coefficients (ICCs) to quantify reliability across different conditions of measurement—for example, scale and interrater reliability quantify generalization across items and raters, respectively. In a multilevel modelling (MLM) context, an ICC is often used to quantify the proportion of a subject‐level variable's variance that is attributable to cluster‐level differences (e.g., between schools or neighbourhoods). When data from cluster members are their ratings of a group‐level characteristic, the ICC is an inter‐rater reliability coefficient, interpreted as the degree to which observations of a cluster generalize across members of that cluster. Structural equation modelling (SEM) is becoming a popular framework for modelling GT, and the ‘omega’ indices of composite reliability—which are functions of SEM parameters—are special cases of GT (i.e., how measurements of a construct generalize across indicators). The merger of MLM and SEM (i.e., ML‐SEM) provides a unique opportunity to explore how well measurements of cluster‐level constructs generalize across multiple measurement conditions (indicators, cluster members, or both). A taxonomy of ICCs is explicated to appropriately quantify reliability across aspects of a procedure for measuring cluster‐level constructs, adding value to the limited ICCs currently considered in ML‐SEM applications.