Caner Sakar

The Optimal Momentum State of Adam under Frozen Local Dynamics

Momentum gives Adam memory: its future is determined not only by the model parameters, but also by the first- and second-moment states inherited from earlier gradients. We ask a conditional question: with the current parameters and adaptive metric fixed, what first-moment state s...

The Optimal Momentum State of Adam under Frozen Local Dynamics

Momentum gives Adam memory: its future is determined not only by the model parameters, but also by the first- and second-moment states inherited from earlier gradients. We ask a conditional question: with the current parameters and adaptive metric fixed, what first-moment state s...

The Optimal Momentum State of Adam under Frozen Local Dynamics

Momentum gives Adam memory: its future is determined not only by the model parameters, but also by the first- and second-moment states inherited from earlier gradients. We ask a conditional question: with the current parameters and adaptive metric fixed, what first-moment state s...