Item wording and the covariance of self-report items: a shape-alignment test, held-out items and an identification limit

Abstract

Working paper, not peer reviewed. If questionnaire items correlate partly because they are worded alike, some of the structure that factor analysis recovers describes the instrument rather than the respondents. Recent studies show that the similarity of item text, measured with language models, tracks the correlations between questionnaire items. This paper puts wording inside a likelihood model of the response covariance as a fixed sentence-embedding kernel, beside a bifactor residual, a nuisance kernel for surface features and a free equal-loading term, and tests it against item-label permutations that refit the whole model. On the DASS-42 (n = 34,583) with three embedding models, the kernel's item correspondence improves the fit far beyond all 100 permutations, under both an all-items and a within-scale null, and it also predicts items left out of fitting. The paper then shows that the share of the uniform (equal-loading) part that is shaped like the kernel is not identified (an exact ridge) from one cross-section in which all items are keyed one way, so that share cannot be read off such data. Scope: all effect sizes are model-scale log-likelihood units, not shares and not calibrated; held-out items share the instrument and respondents; the results are not causal and do not test construct validity. A proposed level-free statistic is ridge-invariant but not identified in general, and no sampler setting qualified for calibration, so no magnitude is claimed. Data: the DASS-42 responses are the Open-Source Psychometrics Project file posted at openpsychometrics.org/_rawdata/; they are not redistributed. The paper reports aggregate statistics only. Code (MIT): https://github.com/abhaymettu/semantic-null-psychometrics