Advanced multi-task learning for brain tumor diagnosis: joint segmentation and classification using attention-guided vision transformers and ensemble models

Abstract

Accurate tumor delineation and reliable tumor subtype classification from multi-sequence magnetic resonance imaging (MRI) is required for accurate diagnosis of brain tumors. Current methods, however, tend to handle segmentation and classification separately with a limited number of features being shared between the two tasks, reduced robustness, and suboptimal clinical applicability. To address these problems, this study presents an integrated multi-task learning framework, which integrates Stability-Guided Multi-View Feature Selection (SG-MvFS), a Cross-Scale Gated Attention Vision Transformer U-Net (CSGA-ViT-UNet), a NeuroViT-XGBoost classification module and a novel ReAL-SAWd optimization strategy. SG-MvFS combines radiomic features, transformer embeddings and clinical covariates to detect stable and discriminative biomarkers. The CSGA-ViT-UNet architecture learns to segment tumors by modeling global contexts and adaptive cross-scale attentions via Vision Transformers, and NeuroViT-XGBoost improves tumor subtype classification by using segmentation-aware representations. Experimental results are reported for the BraTS 2020, BraTS 2021 and BraTS 2023 datasets, showing superior results in comparison to state-of-the-art approaches. The proposed framework outperforms the current state-of-the-art with Dice scores above 97% on BraTS 2020 and 2021 and achieves 92% on BraTS 2023, and AUC scores up to 0.971 with low calibration errors. These results show that the proposed framework can offer an effective solution for achieving joint brain tumor segmentation and classification which is clinically relevant.