Detecting Adversarial Perturbations in AI-Generated Code: A Three-Tier Detectability Hierarchy

Abstract

This study investigates adversarial perturbation detection in AI-generated code. We systematically test five perturbation strategies (comment planting, dead code insertion, variable shadowing, import aliasing, boundary inversion) on 550 AI-generated Python samples (100 clean, 450 perturbed). We evaluate three detection approaches: AST features with XGBoost, mean-pooled CodeBERT with logistic regression, and token-level CodeBERT. Key findings:- Three-tier detectability hierarchy: structural perturbations are trivially detected (100% recall), feature-specific perturbations (import aliasing) require dedicated features (7-17% recall), and semantic perturbations (boundary inversion) are completely invisible (0% recall).- McNemar's test confirms AST features significantly outperform embeddings (p=0.0201).- Cross-strategy generalization reveals asymmetric transferability. Code and dataset: https://github.com/amanmukati09/adversarial-code-perturbations