Multi-level malware subfamily attribution from RGB executable images using attention-augmented deep learning

Abstract

Malware analysis in modern cybersecurity increasingly requires decisions at multiple levels of granularity, including maliciousness detection, malware family identification, and fine-grained subfamily attribution. While image-based malware classification has demonstrated promising performance for binary detection and family-level recognition, the application of RGB executable image representations to hierarchical malware subfamily attribution remains largely unexplored. This study presents a unified hierarchical deep learning framework for malware classification using RGB representations of Windows Portable Executable (PE) files, enabling simultaneous malware detection, family classification, and 33-subfamily attribution within a multi-output architecture. Four attention-augmented transfer learning models ResNet50V2-CBAM, DenseNet169-CBAM, Xception-NonLocal, and VGG19-Triplet Attention were evaluated under identical datasets, taxonomy, preprocessing, and training conditions to ensure fair comparison. Experimental results achieved 99% binary classification accuracy, 96–97% family-level accuracy, and 91–95% subfamily-level accuracy, with macro F1-scores up to 0.89 for fine-grained attribution. A comprehensive experimental evaluation using per-class precision, recall, F1-score, and confusion-matrix analysis showed closely comparable performance among the three strongest attention-based models, with the fully fine-tuned VGG19 variant trailing on the most difficult variants. A protocol-matched comparison with our previously published grayscale framework further showed that RGB executable images provide competitive hierarchical performance while enabling direct compatibility with modern three-channel transfer learning architectures. This study represents one of the first systematic evaluations of RGB executable image representations for hierarchical malware subfamily attribution across a 33-subfamily taxonomy. The proposed framework provides an effective and computationally efficient approach for fine-grained malware attribution, supporting practical applications in digital forensics, cyber threat intelligence, and security operation centers.