The Optimal Momentum State of Adam under Frozen Local Dynamics

Abstract

Momentum gives Adam memory: its future is determined not only by the model parameters, but also by the first- and second-moment states inherited from earlier gradients. We ask a conditional question: with the current parameters and adaptive metric fixed, what first-moment state should Adam carry to minimize cumulative loss over the next H steps of a frozen local model? Under a local quadratic objective, a frozen second-moment preconditioner, and exact finite-time first-moment bias correction, we solve this problem analytically. On positive effective curvature, the unique optimal hidden momentum is an explicit finite-horizon spectral filter of the current effective gradient. The excess cost of any other momentum is a nonnegative quadratic quantity that we call optimizer-state burden. We further show that the momentum-cost Hessian preserves the inertia of the effective spatial Hessian, giving the exact well-posedness boundary of the unrestricted state problem, and that the relevant response polynomials have degree 2H−3. Consequently, fixed-direction burden curvature and derivative quantities admit finite Krylov recovery in exact arithmetic; for H=5, four and seven Hessian-vector products are sufficient, respectively. Exact frozen-metric quadratic benchmarks, including a frozen-feature regression head and a frozen-feature Heat-PINN head, verify the identities to numerical precision. The result is local and finite-horizon: it identifies what optimizer memory should be present under the stated frozen dynamics, rather than claiming global superiority of a modified optimizer on an evolving nonlinear training trajectory.