Receipts, Not Similarity: Grounding Security Explanations of Software Repositories in a Structured, Sourced Knowledge Library. A Study Protocol with a Feasibility Pilot on OWASP ASVS

Abstract

Tools that use large language models (LLMs) to flag security problems in code must also explain them: which requirement is violated, and how to fix it. These explanations are only useful if they cite the right authoritative source. The common way to supply that source is vector retrieval-augmented generation (RAG), which fetches text chunks by embedding similarity. Similarity says nothing about whether a passage is current, who confirmed it, or whether it is the exact clause that applies. We propose instead to store security knowledge as a library in Google's Open Knowledge Format (OKF), where each requirement is one file carrying its source, verification record and expiry, and to let the model navigate the library through its index and links. This paper makes two contributions. First, a feasibility pilot: we converted OWASP ASVS 5.0.0 (345 requirements), the superseded ASVS 4.0.3 (286 requirements) and 152 linked CWE entries into a 783-concept OKF v0.2 library. The library is conformant, reproducible byte-for-byte, and has no broken links among 470. The pilot also exposed a concrete hazard: of the 151 ASVS 4.0.3 requirement identifiers that still exist in 5.0.0, 150 now denote a different requirement. Second, a pre-registered study protocol comparing four retrieval conditions on citation correctness, citation verifiability, use of outdated guidance, and cost. All main-study results are pending and reported as such. Files: archai_okf.pdf (paper); replication_package.zip (build and measurement scripts, and the generated OKF library). The ASVS-derived library in the zip is licensed CC BY-SA 4.0 with attribution to the OWASP Foundation, as required by the ASVS license.