A novel deep learning-based fraud detection framework integrating autonomous PSO-GA-based feature selection, hyperparameter optimization, and explainable artificial intelligence

Abstract

Fraud detection is a critical component of financial security, aiming to accurately distinguish fraudulent transactions from the vast volume of legitimate financial activities. However, this task remains highly challenging because fraudsters continuously evolve their attack strategies to circumvent existing security mechanisms, while the severe class imbalance inherent in transaction datasets further degrades detection performance. Although numerous approaches have been proposed, a robust, accurate, and interpretable fraud detection system has not yet been successfully developed. This paper suggests a novel hybrid deep learning framework for fraud detection as a solution to these problems. The primary contribution of this work is the development of a hybrid framework featuring an autonomous optimization mechanism that dynamically selects between Particle Swarm Optimization (PSO) and Genetic Algorithm (GA) for hyperparameter optimization and feature selection, deep learning (DL) models for fraud classification, and an Explainable Artificial Intelligence (XAI) layer integrating Grad-CAM, SHAP and LIME to provide both global and local interpretations of model decisions. The proposed methodology consists of four main stages: data preprocessing, metaheuristic-based hyperparameter optimization and feature subset selection, transaction classification using DL and machine learning (ML) models, and XAI-based interpretability analysis. The proposed framework was thoroughly tested on the 284,807 transactions of the European cardholder fraud dataset and the BankSim Credit Card Payment Dataset, and contrasted with a number of cutting-edge methods. Experimental results demonstrate that the XAI-enabled PSO-optimized LSTM framework trained on the compact 14-feature subset (α3) consistently outperformed all competing models and achieved the most favorable balance between computational efficiency, predictive performance, robustness, false positive reduction, and interpretability. The proposed model attained an AUC-ROC of 0.9954, an F1-score of 93.65%, a precision of 95.62% and an accuracy of 99.96%, outperforming cutting-edge methods while providing a computationally efficient and more reliable solution for fraud detection.