KVG Medical College & Hospital

Scaffold, Not Seed -- Except One: Positional Encoding and the Emergent Group Representations of Grokking Transformers

A small transformer that groks modular addition famously discovers the Fourier representation of Z/p on its own. Prior work asks what algorithm is learned and which irreducible representations are used, always with the architecture held fixed. We ask one level up the causal chain...