Causality-driven hybrid intrusion detection with entropy-aware feature learning and federated intelligence

Abstract

Modern IDSs should be capable of detecting and recognizing new forms of attack traffic (including both attacks from known adversaries) and to do so in ways that remain computational feasible when deployed across multiple, geographically distributed sites. Rule based, representation learning, and federated approaches have been used separately, or in combination in some cases. However, each approach operates independently in many cases. Therefore, they fail to support causal interpretability, efficient use of feature information, stable performance against adversarial attacks, and generalizability across multiple sites. This paper outlines a causality driven hybrid ids framework. The hybrid framework includes the Causal-Adaptive Signature Learning Engine; the Entropy-Guided Feature Compression Network; the attention-based Hybrid Cognitive Detection Synthesizer; the Adversarial Robustness Calibration Module; and the trust-weighted Federated Threat Intelligence Evolution Framework. The proposed framework has been tested on three separate publicly available network traffic datasets (cicids2017, unsw-nb15, and IoT-23). Testing included: use of dataset specific partitions that controlled for leakage in training; five fold validation to test the ability to learn patterns within the training data; ten different random seed controlled evaluations. A comparison of the hybrid ids framework was made to: block chain-based collaborative ids frameworks; traditional machine-learning based ids frameworks; optimized XGBoost-based ids frameworks. All comparisons were completed under similar preprocessing conditions and distributions of classes. The proposed ids framework produced detection accuracies of 97.4% and 96.1% on the cicids2017 and unsw-nb15 datasets respectively. An overall detection accuracy rate of 95.6% was achieved on the IoT-23 dataset. Additionally, the proposed framework reduced the number of features retained to 18 (a reduction of 70.9%) compared to the original number of features present in each of the datasets. Across ten iterations, the reported aggregate performance metrics were: overall accuracy = 96.37%; precision = 95.97%; recall = 95.43%; F1-score = 95.68%. In addition, the proposed framework had an average processing time per inference of 18.6 milliseconds. Finally, testing performed under bounded gradient-based perturbation conditions indicated that the proposed framework maintained high levels of detection accuracy (>91.5%).