ChatGPT for pulmonary rehabilitation in chronic obstructive pulmonary disease is clinically credible but challenging for patients to use

Abstract

Although interest in applying large language models (LLMs) in healthcare is rapidly growing, their role in supporting pulmonary rehabilitation (PR) for patients with chronic obstructive pulmonary disease (COPD) has not been well established. The primary objective of this study was to evaluate the accuracy, readability, safety, and overall quality of ChatGPT-generated responses through expert assessment. As a secondary objective, we assessed the use of ChatGPT among patients with COPD through information-seeking tasks. A two-round Delphi process with a multidisciplinary expert panel was used to develop consensus-based patient questions regarding PR. ChatGPT-4o generated responses to the final 19 items, which were independently evaluated by experts across four domains: accuracy, readability, safety, and overall quality. In a separate accessibility assessment, 10 patients with COPD performed an information-seeking task using ChatGPT, and their experiences and comprehension were explored. Twenty experts participated, demonstrating high inter-rater agreement (Gwet AC2 = 0.87; 95% CI 0.85–0.89). Duplicate response generations showed high contextual similarity using OpenAI’s embedding model (mean 0.94, SD 0.03; range 0.87–0.98). ChatGPT demonstrated consistently strong performance across evaluation domains (accuracy 3.63; readability 3.57; safety 3.68; overall quality 3.64). However, in the information-seeking task, only 2 of 10 patients were able to retrieve COPD-related PR information using ChatGPT after appropriately contextualizing their condition. Most participants did not provide disease-specific information during the interaction. ChatGPT shows promise as an educational tool for COPD PR, but limitations remain in clarity, contextual specificity, and cultural adaptation. Use of ChatGPT was limited, particularly among individuals with lower digital literacy. These findings suggest that ChatGPT should be implemented with appropriate user guidance and clinician oversight, and that its role in patient education may depend on user proficiency and contextual input.