An explainable biomedical foundation model via large-scale concept-enhanced vision–language pretraining

Abstract

Artificial intelligence for medical imaging is required to be accurate and interpretable to clinicians. However, current multimodal biomedical foundation models often prioritize performance over explainability. Here we present ConceptCLIP, an explainable biomedical foundation model that achieves state-of-the-art diagnostic accuracy while delivering human-interpretable explanations across diverse imaging modalities. We curate MedConcept-23M, a large-scale dataset comprising 23 million biomedical image–text–concept triplets. Leveraging this dataset, we pretrain ConceptCLIP via joint image–text and region–concept alignment for precise and interpretable medical image analysis. Across a large-scale benchmark covering 78 datasets in 10 imaging modalities, ConceptCLIP demonstrates superior diagnostic performance while providing human-understandable explanations. In a clinician user study spanning three modalities, the concept-based explanations provided by ConceptCLIP help clinicians verify model predictions and identify potential errors. As an explainable biomedical foundation model, ConceptCLIP represents a critical milestone towards the widespread clinical adoption of AI, thereby advancing trustworthy AI in medicine. ConceptCLIP is an explainable foundation model that leverages MedConcept-23M, a large-scale dataset comprising 23 million biomedical image–text–concept triplets, for precise and interpretable medical image analysis and diagnostics.