Gating and Normalization Interact in the Attention Block

Abstract

We study how an elementwise sigmoid attention-output gate interacts with normalization placement in small causal transformers (29–36M parameters). A 54-model main sweep crosses three schemes (pre-norm; pre-norm + QK-norm; sandwich + QK-norm), three depths, three seeds, and models with and without the gate on WikiText-2. At the best validation checkpoint selected separately for each run, gating lowers mean validation perplexity in all nine normalization-depth cells. Averaged across depths and seeds, the gated models improve over their matched no-gate baselines by 1.03 PPL with pre-norm, 2.64 with pre-norm + QK-norm, and 3.35 with sandwich + QK-norm. The improvement is therefore 2.32 PPL larger for sandwich + QK-norm than for pre-norm (p < 0.001). A separate fixed-budget 8-layer experiment on WikiText-103 independently finds a gate benefit under all three schemes (all p < 0.01). Normalization placement also produces qualitatively different learned gate distributions: C1 and C2 become low-mean suppressors with essentially no fully-open values, while C3 becomes bimodal, with roughly 20% of values fully closed and 10% fully open. A single-layer ablation identifies the attention-output RMSNorm as necessary and sufficient for this bimodality at similar perplexity, and the distinction persists across the stable learning-rate range. Gate sparsity consistently peaks in the second transformer block.