Confidence Revisited: The Distribution of Information

Abstract

Classical statistics assesses the confidence of a prediction by measuring error variance, estimation variance, or the sum of both. More complex machine learning models assess confidence using empirical methods such as resampling or conformal prediction. Model-based assessments of confidence, however, are likely to be unrealistically tight because they assume that the model is specified correctly and the predictive variables are measured without error. Relevance-based prediction, a model-free prediction technique, assesses the reliability of a prediction from the distribution of information that is used to form a prediction. It gives a more realistic assessment of an individual prediction's reliability because it does not rely on model correctness nor the assumption that the predictive variables are error free.