Sharp KL Convex Aggregation under Arbitrary Misspecification

Abstract

We determine the optimal high-probability excess Kullback–Leibler risk for proper convex aggregation under arbitrary misspecification. The excess is measured relative to the best convex mixture of a dictionary of M densities. For n ≥ 1, M ≥ 2, and failure probability 0 < δ ≤ 1/2, assuming only finite oracle KL risk, the minimax high-probability excess risk is min{log M, log(1 + (M − 1)/n) + log M log(1/δ)/n} up to universal constants. A proper aggregate attains this rate in an oracle inequality with leading constant one, with probability at least 1 − δ, for arbitrary dictionaries of densities, without bounds on density ratios or a common support assumption. The matching lower bound holds on realizable finite-alphabet submodels. Thus arbitrary misspecification does not increase the worst-case minimax order of the excess KL risk. For M ≤ n, we use bounded comparison scores to construct a convex set of mixtures determined by the data that contains the KL projection with high probability. A general localization theorem based on convex volume bounds the discrepancy over this set with a term proportional to dimension, without an additional logarithm in sample size. Continuous aggregation over the localized set then converts the discrepancy bound into an excess KL risk bound. For M > n, we aggregate a fixed finite family formed by combining sparse mixtures with the uniform mixture.