A systematic workflow for utilizing and evaluating large language models in public health policy framing analysis

Abstract

Abstract Background For public health policy research, it is crucial to monitor policy debates to inform evidence-based policymaking. One popular method is to analyze the news for salient frames used to argue for or against a controversial topic. However, this method is time and resource-intensive since domain experts need to manually analyze relevant articles to identify frames. This study presents an approach for using large language models (LLMs) to extract frames and to systematically evaluate results. Methods Building on the results of a framing analysis of the German media debate about a tax on animal-based foods, we developed a systematic workflow to utilize LLMs to extract all instances of the 10 most frequent manually coded negative frames from 46 news articles. With one prompt per article in a few-shot setting we generated a list of quote/frame pairs. We then evaluated results against human-coded ones and calculated precision, recall, and F1-score. Results GPT-5 achieved 79.8% precision and 79.8% recall, identifying 75.2% of manually coded frames. Extracted quotes generally aligned with human-extracted text sections. Gemini 3.1 Pro Preview yielded higher precision but lower recall, resulting in a comparable F1-score (79.0% vs. 79.8% with GPT-5). These findings suggest the approach is generalizable across different frontier models. Conclusions This study introduces a workflow for LLM-assisted frame extraction and transparent evaluation against an expert reference standard. Integrating this approach into public health policy coding can save resources and accelerate analysis. Ultimately, this could increase the representativeness and timeliness of data, enhancing our understanding of ongoing public health policy debates.