A Study of Variability in Brief Assessment of Transactional Success in Aphasia Story Retelling Performance Using Generalizability Theory

Abstract

Purpose: This study used generalizability theory (G-theory) to assess the reliability of the Brief Assessment of Transactional Success in aphasia (BATS) and whether reliability can be maintained with more efficient study designs. Method: G-theory was applied to a data set from BATS test development, where story retells were scored using main concept analysis. Linear mixed-effects models were used to estimate the proportion of variability in scores attributable to storyteller (a person with aphasia [PWA] or familiar/unfamiliar conversation partner [CP]), rater (human or automated), and stimuli. Decision studies (D-studies) were conducted to determine the number of stimuli needed for acceptable reliability. Score distributions and correlations were examined to identify stimuli that disproportionately contributed to measurement error. Results: Differences in performance between PWA contributed most to variability in scores, whereas rater contributed minimally. Stimuli contributed substantially, primarily through interactions with PWA and storyteller. Two stimuli ( Bicycle Boy and Ferguson ) emerged as the least and most challenging; when excluded, stimuli were no longer a significant source of variability. D-studies showed that reliable results could be obtained with three to four stimuli, or as few as two when Bicycle Boy and Ferguson were excluded. CP familiarity contributed to variability in scores via its interaction with PWA. Conclusions: The BATS demonstrated high reliability despite multiple sources of variability in scores. Results support the reliability of automated main concept scoring and show that stimuli-driven error can be controlled with a limited number of stimuli. These results offer guidance for designing efficient, reliable discourse studies.