Scaffold, Not Seed -- Except One: Positional Encoding and the Emergent Group Representations of Grokking Transformers

Abstract

A small transformer that groks modular addition famously discovers the Fourier representation of Z/p on its own. Prior work asks what algorithm is learned and which irreducible representations are used, always with the architecture held fixed. We ask one level up the causal chain: does the positional encoding (PE) -- the one component that explicitly injects structure into the residual stream -- shape the group representation the network grows?We pre-registered the hypothesis that it does not ("PE is scaffolding"), then ran a 5 PE x 5 seed ablation grid (none, learned, sinusoidal, RoPE, shuffled) on grokking Z/59 addition, extended by two direction-control conditions. The null was refuted at exactly one point: learned, rotary, absent, and shuffled PEs leave the emergent representation statistically indistinguishable, while fixed sinusoidal PE compresses it (top-10 Fourier power 0.77 vs 0.53, 8.5 sigma), slows grokking ~1.5x, and impairs recovery from slingshot instabilities.A paired contrast with bitwise-identical initializations shows sinusoidal PE compresses the frequency budget while frequency identity stays lottery-like under every PE. Interventions isolate the mechanism: at sequence length 2 the sinusoidal construction degenerates to a fixed bias of norm 8.0 welded into the residual stream. Scaling the bias yields a dose-response in sparsity (Spearman rho 0.76); centering it while preserving the positional difference exactly eliminates every symptom; a frozen random PE of matched norm but different direction reproduces every symptom. The culprit is the fixed large-norm bias -- not periodicity, not direction. On S5, whose solution needs high-dimensional irreducible subspaces, the identical bias does no measurable harm: the pathology requires a low-dimensional target. PE is scaffolding as a rule; the exception is a construction artifact with a one-line fix.