Shubham Kumar

Gating and Normalization Interact in the Attention Block

We study how an elementwise sigmoid attention-output gate interacts with normalization placement in small causal transformers (29–36M parameters). A 54-model main sweep crosses three schemes (pre-norm; pre-norm + QK-norm; sandwich + QK-norm), three depths, three seeds, and models...

Gating and Normalization Interact in the Attention Block

We study how an elementwise sigmoid attention-output gate interacts with normalization placement in small causal transformers (29–36M parameters). A 54-model main sweep crosses three schemes (pre-norm; pre-norm + QK-norm; sandwich + QK-norm), three depths, three seeds, and models...