Deep learning framework for breast cancer subtypes detection and diagnosis

Abstract

Molecular subtyping of breast cancer serves as a cornerstone for personalised diagnosis and therapeutic monitoring; nevertheless, the deployment of advanced deep learning models is often hindered by the lack of accurate tumour segmentations in extensive datasets and the intrinsic class imbalance among molecular subtypes, particularly the under-representation of the HER2-enriched subtype in public archives. This study introduces a comprehensive and practical pipeline for characterising breast cancer phenotypes, effectively addressing these challenges. We have collected publicly accessible images belonging to breast cancer from different datasets using magnetic resonance imaging, encompassing categorised molecular subtypes, with or without manual tumour segmentations. Using a 3D Convolutional Neural Network (3D CNN) model trained exclusively on the Duke Breast Cancer MRI dataset, which contains bounding box annotations and molecular subtypes, we developed a robust classifier. To extend our analysis to data without annotations, we applied a Faster R-CNN model, pre-trained on the same Duke dataset, to generate lesion detections in the remaining multicentric datasets. From these, we strategically selected a balanced subset of 200 cases (50 for each phenotype: Luminal A, Luminal B, HER2-enriched and Triple-Negative), based on the highest detection confidence, thus solving the class imbalance problem and ensuring equitable representation of all subtypes for rigorous testing of our classifier. The final validation was performed on an independent clinical dataset from a public hospital in Setúbal, Portugal, where the model demonstrated strong discriminative performance with AUC-ROC values ranging from 0.8786 to 0.9825 across the four molecular subtypes, providing promising evidence of cross-institutional generalisation under a pronounced subtype prior shift between the North American training distribution and the Portuguese evaluation cohort. This work, complemented by a data anonymisation system and close collaboration with clinical specialists, establishes a scalable and effective methodology for precision radiogenomics that simultaneously overcomes the challenges of data imbalance and the absence of detailed annotations. Further multi-site prospective validation on larger and ethnically diverse cohorts is required before clinical deployment.