Quiet or Blind? Measuring the Cost of Exclusions in LLM-Generated Detection Rules

Abstract

A detection rule costs its operator twice, in analyst time for every benign alert it raises and in dwell time for every attack it misses, and the exclusion clause that encodes what is normal in one environment decides both. Language models now write such rules from threat reports, and prior work judges them by text similarity or by the attacks they catch, neither of which can see a rule that is quiet because it is blind. Here we execute the rules instead. Fifteen model configurations from seven organizations, each pinned to one endpoint and one token budget, and to one seed where the endpoint accepts it, under a frozen pre-registration, write Sigma rules for 238 SigmaHQ requirements. Every rule then runs as written, with its exclusions removed by an AST-safe transform, and beside the human rule, on 384,191 benign Windows events and 395 attack files. Removing exclusions from 622 human SigmaHQ rules multiplies benign alerts 29x (196 to 5,713), and the generated rule is louder than the human on 87 of the 90 untied live comparisons on the matched requirements. Generated rules carry exclusions on 20% to 83% of the rules they produce; on the 30 requirements the benign corpus can exercise, two in five of those exclusion-bearing rules are inert, two thirds of the rest leak, and over all 1,450 exclusion-bearing rules 92% never fire. On the attack corpora the model's own exclusion is the sole reason for a miss in 99 of 1,018 exclusion-carrying cases (9.7%, 95% CI 6.1-14.0%), and 345 of 1,164 quiet generated rules (30%) are blind, covering fewer than half of the attack files the human rule catches. An arm's silence on benign data is positively rank-correlated with its blindness on attack data (Spearman rho = 0.77 over fifteen arms). Given the human's exclusion values to choose from, every configuration recovers more than it generates. A false-positive rate is therefore not interpretable without a silence rate beside it, and among the measurements reviewed here only a coverage test on attack logs separates quiet from blind. We release the harness, prompts, raw logs and pre-registrations to run it.