Linguistic inequity in clinical AI evaluation: a scoping review of 806 studies across 62 languages

Abstract

Abstract Background Large language models are increasingly used clinically in many non-English languages, yet most safety and performance evidence comes from English. Whether they are safe in the languages of the populations most exposed to them is largely unknown. We aimed to map the evidence base for non-English clinical evaluation and assess how well it matches population and disease burden. Methods We conducted a scoping review of PubMed, Scopus, and Embase (January 2022 to February 2026) and identified 806 studies evaluating large language models on clinical tasks across 62 non-English languages. We characterised their linguistic and geographic distribution, clustered designs into methodological archetypes, modelled archetype membership using multinomial logistic regression, and scored reporting completeness against language-specific items. Results Here we show that non-English clinical evaluation concentrates in a few high-resource languages. The African and South-East Asian regions (3.15 billion people) contribute 17 of 806 studies (2.1%), with no local-language evaluation for their leading causes of death. Mid-resource languages are more likely than high-resource languages to appear in multi-model examinations rather than clinical utility studies (relative risk ratio 1.79, 95% confidence interval 1.13 to 2.84). Four language-specific items are rarely reported (median 0 of 4) and do not improve over time. Conclusions The evidence base for clinical language-model safety is misaligned with global deployment and disease burden. We propose LINGUA, a candidate reporting framework that extends existing guidelines to non-English clinical evaluation, and conclude that closing the gap requires external reporting standards and funding priorities, not the passage of time.