The Spectrum, Not the Norm: Noise and Inexactness in Orthogonalised Steepest Descent

Abstract

Orthogonalised momentum methods - which replace a matrix-shaped update by the polar factor of its momentum buffer - have become competitive with Adam for training transformers, and are exactly steepest descent in the spectral norm. This paper asks what actually controls their behaviour under the two departures from theory that every implementation makes: the orthogonalisation is inexact, and the gradient is noisy. We give three results. First, a stable-rank conversion: the natural nuclear-norm stationarity guarantee is worth a factor r* in the usual Frobenius measure, so the method's advantage over Euclidean geometry is exactly the ratio r*/kappa of gradient stable rank to the curvature ratio kappa=L_op/L_F in [1,min(m,n)]; we measure both quantities on trained transformers. Second, an inexact-oracle theorem: an update with >= (1-d_-)||G||_* and ||O||_op <= 1+d_+ retains a ((1-d_-)/(1+d_+))^2 fraction of the guaranteed progress. Measuring (d_-,d_+) for the standard five-step Newton-Schulz iteration shows its efficiency is non-monotone in the number of steps, and depends strongly on the gradient spectrum. Third, and most consequentially, we prove >= ||G||_*-2||N||_* and then show this bound, though tight in the worst case, is badly pessimistic for isotropic noise: a refined inequality >= ||G||_*-||N||_op ||O(G+N)-O(G)||_* identifies the operative quantity as the noise operator norm measured against the signal's singular-value spectrum. Empirically, a flat spectrum retains 94% alignment even when ||N||_*=4||G||_*, while a 1/i spectrum falls to 49%. We then measure all of this on the gradients of a pretrained 0.5B transformer: stable rank averages 86.3, about 9.6% of min(m,n); the nuclear-norm bound is negative - and so vacuous - for 100% of the matrices tested; and a one-constant "noise-floor" model, calibrated on synthetic matrices and applied here without refitting, predicts the measured alignment to an RMSE of 0.080 against the bound's 4.487. Finally, the nuclear-norm view predicts that the momentum time-scale must grow with width; we test that in a controlled quadratic and in character-level transformer training, and report that it fails in both.