AI and Machine Learning in Automated Penetration Testing: A Literature Review

Abstract

Automated penetration testing using artificial intelligence and machine learning has emerged as a promising alternative to traditional, expertise-dependent manual testing. This paper presents a literature review examining AI/ML-driven approaches to automated penetration testing across three dimensions: the underlying tools and techniques, their reported effectiveness, and the challenges limiting practical deployment. The review identifies two dominant paradigms — reinforcement learning, which excels at precise decision-making within well-defined attack-path spaces, and large language models, which offer flexible, context-aware reasoning but are prone to losing context over extended, multi-stage tasks. While early framework-specific studies report strong performance improvements, more rigorous stage-level benchmarks reveal substantially weaker results under standardized evaluation conditions, exposing a gap between reported capability and real-world readiness. The review further identifies the absence of standardized benchmarking practices, unresolved integration between reinforcement learning and language-model-based approaches, and limited attention to accountability and safety concerns as key directions for future research. These findings suggest that advancing automated penetration testing will require rigorous evaluation methodology as much as technical innovation.