Multistage advanced persistent threat detection and malware attribution using Mamba state space models with Kolmogorov Arnold networks

Abstract

Advanced Persistent Threats (APTs) represent the most sophisticated class of cyberattacks, characterized by multi-stage, stealthy, and persistent intrusion campaigns orchestrated by well-funded, often state-sponsored adversaries. Existing detection methods suffer from three critical limitations: inability to model long-range temporal dependencies across sequential attack phases, reliance on fixed activation functions that cannot capture subtle decision boundaries between stealthy APT traffic and normal operations, and lack of interpretability in detection decisions. In this paper, we propose MambaKAN-APT, a deep learning framework that synergizes Kolmogorov–Arnold Networks (KAN) with Mamba Selective State Space Models (SSMs) for multi-stage APT attack detection and malware family attribution. The KAN Feature Encoder employs learnable edge-wise activation functions parameterized as B-splines (evaluated by the De Boor recursion) to automatically discover nonlinear feature transformations for each input dimension. Stacked Mamba SSM blocks implement a genuine selective state-space recurrence that captures temporal dependencies across sequential attack phases with linear computational complexity \({\mathscr {O}}(L)\) , unlike the quadratic cost of Transformer architectures. A dual-pooling classification head combined with Focal Loss addresses the severe class imbalance inherent in APT datasets, while SHAP-based explainability provides interpretable per-feature importance analysis. We evaluate MambaKAN-APT on two real, publicly available benchmarks under a rigorous, leakage-aware, reproducible protocol with results reported as mean ± standard deviation over repeated runs: the DAPT 2020 network-flow benchmark (86,691 flows, 64 informative CICFlowMeter features across five attack phases, with Normal traffic forming 73.5% of flows) and the CyberScienceLab APT malware dataset (1,428 samples across five threat-actor groups, represented by disassembled opcode sequences). Because DAPT 2020 is highly imbalanced, we adopt macro-F1 as the primary metric. On the tabular DAPT 2020 task, gradient boosting (XGBoost) is a very strong baseline (macro-F1 0.758) and is competitive with or stronger than the deep models on macro-F1; we report this explicitly. On the genuinely sequential CyberScienceLab attribution task—the setting best suited to a state-space model—MambaKAN-APT attains the highest mean macro-F1 ( \(\approx 0.959\) ) with low across-seed variance, although on this balanced task it is statistically tied with the strong XGBoost n -gram baseline ( \(\approx 0.954\) ; paired t -test p = 0.59), a result we report transparently rather than over-claiming a sequential advantage. To the best of our knowledge, this is the first work combining Mamba SSMs with KAN architectures for APT detection. SHAP analysis identifies flow- and packet-level statistics (DAPT 2020) and discriminative opcode n -grams (CyberScienceLab) as the most influential features. All code, cached features, fixed seeds, and download scripts are deposited for full reproducibility.