Hybrid rank fusion of a group-token transformer with gradient boosting improves detection of evasive and novel-family malware

Abstract

Static machine-learning malware detectors report near-perfect closed-set accuracy, but the malware that matters operationally is the malware that slips past detection and the malware that belongs to families never seen in training. We study both on three longitudinal benchmarks: EMBER2024 (3.2 M Windows, .NET, APK, ELF and PDF files, 52 training weeks and 12 test weeks, plus a challenge set of 6,315 files that evaded every antivirus engine at first sight), LAMDA (1.0 M Android applications, 2013–2025, with an IID/NEAR/FAR drift protocol) and BODMAS (134 K PE files, 2019–2020, monthly evaluation). We make five contributions. First, an operational evaluation protocol: true-positive rate at 1% and 0.1% false-positive rate, challenge-set detection at fixed FPR, novel-family, singleton and low-consensus strata, week-by-week and year-by-year drift, thresholds calibrated on time-aware validation data, and paired-bootstrap confidence intervals. Second, three new deep architectures for static feature vectors, trained jointly across formats: GT-Former, a Group-Token Transformer over the semantic feature groups, whose hyper-parameters are tuned with Optuna on a time-aware validation window and which is evaluated with five seeds on every benchmark; FR-MoE, a format-routed mixture of experts; and ProtoCon-Net, a prototype classifier with family-aware supervised-contrastive training. Third, three metadata-driven objectives that use antivirus consensus and family labels at training time only (a consensus-aware margin loss, a consensus-regression head and a family-adversarial gradient-reversal head), with full ablations. Fourth, explainability with SHAP and CLS attention. Fifth, benchmark-hygiene findings, including that the public EMBER2024 release lists every record twice. The results are sobering and useful: the EMBER2024 LightGBM benchmark remains the strongest single model at strict operating points (Win32: 90.0% TPR at 0.1% FPR versus \(88.1 \pm 0.4\) % for the tuned GT-Former over five seeds), and the metadata-driven objectives do not pay for themselves: on LAMDA, with five seeds per variant, no objective activated on its own is distinguishable from the plain model, and only their combination degrades the strictest operating point (NEAR TPR at 0.1% FPR falls from \(10.1 \pm 2.3\) % to \(4.4 \pm 3.7\) %). The seed spread on this benchmark is large enough that single-seed ablations are unreliable, which we regard as a methodological finding in itself. The deep models are nevertheless complementary to gradient boosting: a rank average of the tuned GT-Former and LightGBM raises Win32 challenge-set detection from 63.2% to 68.8% at 1% FPR (95% CI \(+4.7\) to \(+6.6\) points), novel-family detection from 89.1% to 91.8% (CI \(+1.5\) to \(+3.9\) ) and test TPR at 1% FPR from 96.8% to 97.3%, with significant challenge-set gains also on .NET ( \(+1.6\) ) and APK ( \(+3.9\) ); on LAMDA a rank average of an MLP and LightGBM raises far-future (2018–2025) detection from 40.5% to 54.4% at 1% FPR (CI \(+13.7\) to \(+14.0\) ) and singleton-family detection from 34.6% to 53.3% when the threshold is calibrated on the benign files of the test region ; under a threshold fixed on validation data and applied unchanged, twelve years of drift move both models far below their nominal budget and the advantage falls to 0.1 points (Section Retrospective and deployment operating points), which we regard as the honest deployment figure; and on BODMAS the tuned GT-Former/LightGBM fusion matches LightGBM at 1% FPR (99.85%) and improves detection at 0.1% FPR (98.9% to 99.5%) and novel-family detection at 0.1% FPR (94.7% to 97.1%). SHAP shows why the benchmark misses evasive files: its strongest evidence is structural sloppiness (suspicious section flags, missing signatures), and evasive malware is exactly the malware that avoids it. Code, de-duplication masks, models and all score files are released.