Concerning uncertainty—a systematic survey of uncertainty-aware XAI

Abstract

This paper surveys uncertainty-aware explainable artificial intelligence (UAXAI), examining how uncertainty is incorporated into explanatory pipelines and how such methods are evaluated. Across the literature, three recurring approaches to uncertainty quantification (UQ) emerge (Bayesian, Monte Carlo and conformal methods), alongside distinct strategies for integrating uncertainty into explanations: assessing trustworthiness, constraining models or explanations and explicitly communicating uncertainty. Evaluation practices remain fragmented and largely model centred, with limited attention to users and inconsistent reporting of reliability properties (e.g. calibration, coverage and explanation stability). Recent work leans towards calibration, distribution-free techniques and recognizes explainer variability as a central concern. We argue that progress in UAXAI requires unified evaluation principles that link uncertainty propagation, robustness and human decision-making, and highlight counterfactual and calibration approaches as promising avenues for aligning interpretability with reliability. This article is part of the theme issue 'Advancing uncertainty quantification in AI systems'.