Bo Jiang

New Results on the Polyak Stepsize: Tight Convergence Analysis and Universal Function Classes

Abstract. In this paper, we revisit a classical adaptive stepsize strategy for gradient descent: the Polyak stepsize (PolyakGD), originally proposed in Polyak [ USSR Comput. Math. Math. Phys., 9 (1969), pp. 14–29]. We study the convergence behavior of PolyakGD from two perspectiv...