Agents Don't Game the Linter: Measuring Whether LLM Repair Satisfies the Analyser or Fixes the Defect

Abstract

Four code-specialised LLMs were shown static-analysis findings on their own solutions and asked to fix them. The study was designed to catch Goodharting: satisfying the analyser that was shown while the defect stayed put. It did not happen. Across 4,439 repair steps the models wrote five suppression directives in total, and 94% of the findings they removed were also gone from a held-out analyser they never saw (suppression rate 6%, 95% CI [4%, 9%]). They complied instead, and complying was expensive. Each task ran two repair arms branching from the same baseline solution: one shown Pylint findings, one shown Ruff findings. Restricted to tasks where both arms ran and the baseline passed its own tests, the Ruff arm broke the program where the Pylint arm did not on 46 occasions against 3 the other way (exact McNemar, p = 7 × 10⁻¹¹, 213 pairs, every model individually significant). That result is one rule. Ruff's S311 ("pseudo-random generators are not suitable for cryptographic purposes") broke 41.8% [32%, 52%] of the working programs it appeared in, against 8.8% [5%, 16%] where no security rule was shown. Remove every S311 task and the paired comparison falls to 10 versus 3, p = 0.09. This is therefore not evidence that security rules as a class are dangerous in a repair loop; it is evidence that S311 is, and that one rule was enough to make an entire ruleset look four times more destructive than another. None of the affected code was cryptographic. BigCodeBench uses random for data generation and seeds it in the tests, so the agent swaps in secrets, determinism is lost, and since secrets has no seed() some programs stop running at all: a true positive about the code and a false positive about the context, with the analyser reporting success either way. A suppressed finding leaves a directive in the diff; faithful destructive repair leaves nothing to find. That is an empirical case for verification independent of the gate being optimised against, and a different case from the one usually made. Method: four models from four families (Qwen2.5-Coder-7B-Instruct, deepseek-coder-6.7b-instruct, Yi-Coder-9B-Chat, granite-8b-code-instruct-128k), bfloat16 on one A100 via vLLM at temperature 0; 300 BigCodeBench tasks filtered to those whose reference solution passes its own tests in the run environment; two repair rounds per arm; 4,439 steps.