Normalization as Query: The Geometric Essence of Neural Network Representations

Abstract

Normalization layers (LayerNorm, BatchNorm, RMSNorm) are commonly regarded as engineering tricks for stabilizing training. This paper proves that normalization layers mathematically enforce a decomposition of hidden states into direction and magnitude, thereby establishing the Normalized State Decomposition (NSD) framework: s equals h divided by the norm of h, serving as the query vector; the norm of h is the confidence; and the LM Head weights W form a queryable conceptual memory bank. Based on this decomposition, we verify five coding laws across 4 architectures (Transformer, DeltaNet hybrid, SWA Attention, MoE), 4 scales (1.5B to 26B), and 2 precisions, and develop a series of concept-level operational tools requiring no additional training: Fisher discriminant ratio identifies 254 concept-discriminative dimensions (zero training); causal intervention edits s along concept directions achieving 84% to 100% target logit rank 1; and orthogonal complement projection significantly reduces directional overlap on 93% of class pairs. Readout head probing experiments (WikiText-2, frozen backbone) show that MLP x2 with only 9.44 million parameters (4% of the backbone) reduces perplexity by 29.8%, demonstrating that linear LM Head readout is lossy. We also analyze the complementary relationship between SAE and normalized state decomposition at the concept level, and revise the weight clustering mechanism to coefficient-selective accumulation. The normalized state decomposition framework is universally applicable to all architectures containing normalization layers and linear output heads.