YingXu Wang

Normalization as Query: The Geometric Essence of Neural Network Representations

Normalization layers (LayerNorm, BatchNorm, RMSNorm) are commonly regarded as engineering tricks for stabilizing training. This paper proves that normalization layers mathematically enforce a decomposition of hidden states into direction and magnitude, thereby establishing the No...