A new dataset and comparisons of AI designs helped one system detect more vulnerabilities in codebases it had not seen before.
Researchers examined how model design, settings and training-data quality affect an AI system’s ability to identify software vulnerabilities in unfamiliar codebases. They created VulGate, a dataset that removes mislabeled and duplicate examples, adds newer vulnerabilities and metadata, includes difficult examples, and provides dedicated test sets.
In experiments comparing several AI model designs, encoder-based models performed better in accuracy and generalization than decoder-based models. The researchers’ model reported a 6.8% improvement in recall on BigVul and performed better than other models on unseen projects.
Evidence and caveats
The conclusions come from experiments using the newly introduced VulGate dataset, the BigVul benchmark and tests on unseen projects, including comparisons among encoder-based and decoder-based models. The abstract does not report the dataset sizes, the exact experimental settings, the comparison baseline behind the 6.8% recall improvement, or the detailed results for each unseen project. It therefore does not establish how well the findings would transfer to all software languages, vulnerability types or real-world development settings.
