Canonicalizing the Wrong Thing: Quotient-Aligned Gauge Fixing for Semantic Representations

Abstract

Preprint, version 1. Modern predictive models often admit a representation gauge: optimization, normalization, or post-processing may select a stable representative without selecting the distinctions required by downstream use. We formalize quotient alignment: a selection criterion is semantic only when its minimizers on a prediction-equivalent orbit differ solely by transformations that preserve a declared downstream interface. This separates the familiar question of whether symmetry is broken from the sharper question of whether the right symmetry is broken. We establish both negative and constructive results in controlled operator models. First, for arbitrary linearly independent physical generators in dimension d \ge 2 and invertible action scrambling, exact factor-balance conservation admits a continuum of prediction-equivalent, semantically distinct bases. Thus an optimization statistic can be informative yet quotient-misaligned. We then instantiate structural gauge fixing (SGF) in three settings whose semantics are defined by different invariants: unique rank-one rays, the bracket-derived ideal of se(2), and unique graph-support rays in a constant-rank, non-Lie span. Under the corresponding quotient-aware metrics, the matched criteria expose the declared quotient with residual 10^{-7} or smaller, while whitening, operator PCA, rank, and mismatched Lie canonicalizers can satisfy their own criteria without recovering it. Under component- or block-constrained world changes, matched SGF approaches unrestricted adaptation, whereas wrong gauges do not; unrestricted refitting removes the gap. Two nonlinear bridges then preserve a predictor exactly while changing its native interface. In a locked Transformer, functional SGF reduces median basis error from 0.400 to 3.39\times10^{-8} and raises one-row adaptation accuracy from 51.3% to 100%. On real handwritten images, four gauges have identical attribute predictions to 8.81\times10^{-8} relative error, yet one-row changed-task accuracy ranges from 70.7–81.4% for wrong canonicalizers versus 97.0% for interface-matched SGF. SGF is a representative-selection principle, not universal semantic discovery: it fixes a gauge only after the relevant structure has been specified.