The Hidden Geometry of Transformer Weights: A Canonical Basis for Interpreting Transformer Language Models

Abstract

Version 2 of the paper "The Hidden Geometry of Transformer Weights". Rotating a Transformer language model into a coordinate system alignedwith its own weight matrices — the canonical basis — reveals a hiddeninternal structure. Version 1 established the descriptive phenomena.Version 2 establishes that they are functional and general: - Causal evidence: zeroing a single canonical axis (0.11% of the model) collapses MMLU from 47.50% to 21.25% and destroys output coherence; five control axes show no effect.- Six per-layer spectral indices (cohesive, torsional, informational, dimensional, rhythmic, vorticity) with effective dimensionality 4.77/6, exposing a 41x isotropic collapse between weight and activation spectra.- Lossless realignment verified on RMSNorm and LayerNorm architectures (Qwen 2.5 0.5B, SmolLM2 1.7B, Pythia 1.4B), plus native cross-layer alignment measurements across eight architectures and a MoE model.- Corrected cross-layer alignment numbers vs. version 1. Every measurement is reproducible: scripts and pre-computed dataaccompany the paper (https://github.com/todotge/canonical-basis),including an interactive per-axis control chat(demo video: https://youtu.be/WOJwkjj9VT0).