Human-in-the-loop large language model framework for identification of cutaneous immune-related adverse events

Abstract

This study evaluated a retrieval-augmented, multi-agent large language model (LLM)–driven, human-in-the-loop framework for detecting cutaneous immune-related adverse events (cirAEs) from clinical notes. Compared with the unassisted manual-review group, the LLM-assisted group demonstrated higher accuracy (F1 = 0.88 vs 0.77) and inter-rater agreement measured by Cohen’s κ ( κ = 0.82 vs 0.50), and reported an approximately one-third shorter average review time. This framework pilots how LLMs can be applied to identify immune-related toxicities and, more broadly, support accurate, scalable, and transparent adverse event data extraction.