VSAT: Estimating the Completeness of Multi-Lens LLM Security Audits of Web Applications via Capture-Recapture

Abstract

VSAT (Vulnerability Saturation Auditing) makes the completeness of an LLM security audit a measurable, calibrated quantity. It runs several deliberately diverse LLM audit "lenses" over the same codebase and treats each lens as a capture occasion, so the overlap structure yields (i) a far more complete union vulnerability list and (ii) a Chao2 richness estimate of the undiscovered population. VSAT combines this statistical discovery saturation with a deterministic OWASP ASVS structural coverage into a single completeness score (Phi = C_struct x C_hat) and derives a saturation-based stopping rule. Implemented as a security-audit skill on the cc-rsg-web agentic platform, VSAT attains 94.7-100% category recall and 99.1% code-verified precision on three documented benchmark applications (NodeGoat, django.nV, DVWA), delivers a 2.4-3.4x discovery uplift over a single pass with a per-finding proof-of-concept and regression test, and its non-zero residual estimate is corroborated by an independent real-world field validation. To our knowledge this is the first method to bring capture-recapture completeness estimation and a saturation stopping rule to LLM-based web-application security auditing. Reproduction artefacts (metrics, incidence matrices, analysis scripts, injection answer keys, and matcher) added in this version. v3 (2026-09-24): estimator switched to the canonical incidence Chao2 form with the (T-1)/T factor and 95% log-normal confidence intervals; per-run incidence matrices for all 26 injection-study runs added; run provenance (which corpus each study audited) documented; recompute.py regenerates every reported table from the shipped files without network access.