Forward-KL Learning Curves of Maximum Likelihood Estimators Need Not Be Monotone

Abstract

Let L_n be the expected forward Kullback–Leibler risk of the maximum-likelihood estimator based on n independent observations from a correctly specified model. We show that L_n need not be nonincreasing, even for a one-dimensional full, regular, minimal natural exponential family. For every integer M at least 2, analytic estimates for a compound-Poisson family show that L_1 through L_M form a strictly increasing sequence; Gaussian smoothing preserves quantitative gaps and produces strictly positive carrier densities that extend to entire functions. For a separate two-Gaussian family, 128-bit outward-rounded Arb arithmetic certifies L_21 strictly greater than L_20 at an exact rational parameter point, and continuity extends the inequality to a nonempty open subset of the two-parameter domain.