A multi task deep learning framework for automated health condition prediction from retinal fundus images using hybrid attention and explainable AI

Abstract

Abstract Although retinal fundus imaging is a non-invasive tool to assess systemic and fundus pathology, current deep learning algorithms are mostly single-task, lack clinical interpretability and are unable to handle acquisition variation between different domains. These constraints limit their clinical utility: when a patient comes to the clinic for fundus screening, they could also be diabetic, have glaucoma, AMD, and increased cardiovascular risk, with each disease having retinal vascular biomarkers that require a patient to be tested for in the clinic, which must be managed within a framework of common, interpretable, and generalizable models rather than a pipeline of independent models. To solve these problems, we suggest a multi-task deep learning network that simultaneously classifies DR severity (five-class), glaucoma likelihood (binary), AMD stage (four-class) and cardiovascular risk (binary) from a single retinal fundus image using a single forward pass. It adopts an EfficientNet-B4 convolutional backbone pretrained on the ImageNet dataset, a six-layer multi-head self-attention Transformer encoder (d_model = 512, h = 8), a novel dual-branch spatial-channel attention module that builds on, but significantly expands, the CBAM module, and four task-specific output heads that are optimized using the homoscedastic uncertainty-weighted multi-task loss following the formulation of Kendall et al. Per-task anatomical explanations are produced by back propagating the CNN backbone via Grad-CAM in the anatomical space. The proposed model was evaluated on EyePACS, APTOS 2019, Messidor-2 and RIM-ONE v3 and achieved an accuracy of 94.7%, AUC = 0.978, sensitivity = 93.2% and specificity = 95.8% while being statistically better than all single task models (DeLong test, p < 0.01, Bonferroni corrected). Under the reported experimental setting, the zero-shot cross-dataset evaluation achieves AUC ≥ 0.951, which is promising for its performance across datasets. Grad-CAM saliency maps are anatomically similar to established clinical grading criteria for each disease. The four task-specific heads share a single architecture and produce four outputs in one forward pass, but no individual patient in the training corpora had verified ground-truth labels for all four conditions simultaneously; the model is trained on partially labelled, multi-source data via a masked multi-task loss, and the cardiovascular output is an AVR-derived image-based surrogate rather than a validated clinical cardiovascular diagnosis. This proposed framework could be considered for an additional potential step toward the future when it comes to multi-disease fundus screening with an interpretable single-model solution, if it is assessed and approved at an independent multi-centre level and by regulatory bodies.