Synthetic data assurance with conformal prediction

Abstract

In this paper, we apply the methods of conformal prediction to the task of synthetic data assurance using images augmented with synthetic weather effects, and examine their fidelity and diversity with respect to both clear weather and adverse weather real image data. It can be expensive and time-consuming to collect enough data to train a machine learning (ML) model so that it makes accurate predictions in all contexts in which it is applied, and this has led to much interest in the use of synthetic data. Good-quality synthetic data must be similar to, but sufficiently different from any available real-world data in order to add value to model development; various methods for synthetic data assurance have been proposed. Such methods frequently do not offer performance guarantees on the quality of their output or focus on narrowly defined metrics. With conformal methods, we find that our adverse weather real data and synthetic augmentations are not sufficiently similar, indicating a requirement for improved methods of synthetic augmentation. Conformal approaches provide a holistic view of synthetic data quality, enable curation of synthetic datasets with theoretical guarantees and are generally applicable, making them a valuable tool in synthetic data assurance. This article is part of the theme issue 'Advancing uncertainty quantification in AI systems'.