Reading Between the Lines: A Scoping Review of Large Language Models in Conversational Data Analysis

Abstract

Transformer-based large language models are becoming increasingly popular in supporting qualitative analysis, but their applications and validation are often poorly characterized. This scoping review followed PRISMA-ScR rules to understand how such models are used in the analysis of interview and conversational data. Searches of ProQuest, Web of Science and IEEE Xplore (2020–2025) returned 3,102 records; 207 were assessed at full text, and 84 peer-reviewed studies were included. Support for thematic analysis was the most common application (43/84, 51%), typically with full transcripts analyzed using zero-shot prompting. Decoder-only models dominated (69/84, 82%), with GPT-4 (51%) and GPT-3.5 (21%) most frequent and Llama the leading open-source alternative (10%). Validation, when present, relied mainly on human-LLM comparison ( n = 33) or repeated model runs ( n = 12). Trends regarding reporting model versions and prompts were inconsistent. Standardized reporting and ethical guidance are needed to support reproducible use.