Conformal risk control for non-monotonic losses

Abstract

Conformal risk control is an extension of conformal prediction for controlling risk functions beyond miscoverage. The original algorithm controls the expected value of a loss that is monotonic in a one-dimensional parameter. Here, we present risk control guarantees for generic algorithms applied to possibly non-monotonic losses with multi-dimensional parameters. The guarantees depend on the stability of the algorithm-unstable algorithms have looser guarantees. We give applications of this technique to selective image classification, false discovery rate and intersection-over-union control of tumour segmentations and multi-group debiasing of recidivism predictions across overlapping race and sex groups using empirical risk minimization. This article is part of the theme issue 'Advancing uncertainty quantification in AI systems'.