The Optimal Momentum State of Adam under Frozen Local Dynamics

Abstract

Momentum gives Adam memory: its future is determined not only by the model parameters, but also by the first- and second-moment states inherited from earlier gradients. We ask a conditional question: with the current parameters and adaptive metric fixed, what first-moment state should Adam carry to minimize cumulative loss over the next H steps of a frozen local model? Under a local quadratic objective, a frozen second-moment preconditioner, and exact finite-time first-moment bias correction, we solve this problem analytically. On positive effective curvature, the unique optimal hidden momentum is an explicit finite-horizon spectral filter of the current effective gradient. The excess cost of carrying any other momentum is a nonnegative quadratic quantity that we call optimizer-state burden. We further show that the momentum-cost Hessian preserves the inertia of the effective spatial Hessian and that the response polynomials have degree 2H−3, giving finite Krylov recovery of directional quantities. On a stationary nonlinear Burgers PINN, trust-limited H=5 state interventions derived from the local model reduce the actual nonlinear five-step objective in all nine tested seed/checkpoint conditions, while trust-region-scale response curves closely follow the frozen-local quadratic prediction. A diagnostic control with exactly matched post-step parameters and second moment, but a different first moment, also favors the intervened state in all nine conditions. These experiments support the hidden-state mechanism and a trust-limited local interpretation; they do not claim that the unrestricted local optimum is globally optimal on nonlinear training trajectories.