When Ground Truth Is the Bottleneck: How Self-Report Label Properties Constrain Multimodal User-State Estimation in Human–Robot Collaboration

Abstract

Self-report instruments such as the NASA-TLX, STAI, and SAM routinely serve as ground-truth labels for multimodal user-state estimation, yet their measurement properties are rarely examined as constraints on prediction. In the MultiPhysio-HRC dataset (52 participants, 10 task conditions, 11 self-report variables), a generalizability study decomposed each label's variance into person, task, person-by-task, and error components, followed by leave-one-subject-out prediction. Task-level variance ranged from 9.7% to 62.9%. Treated as a dependability coefficient for the task-induced state, this proportion closely ordered cross-subject prediction (ρₛ = .900 for a physiological model; .982 with task identity added). Physiology improved all 11 labels (mean Δρ = +0.081). Labels fail for distinguishable reasons: person-dominated labels stay weak despite high repeatability, error-dominated labels are unstable at a single administration, and interaction-dominated labels may need person-specific calibration. The resulting limit is termed a measurement-and-construct-fit ceiling: a design-level constraint, not a bound on all predictors.