Dynamic evaluation of uncertainty quantification under distribution shift in materials property prediction

Abstract

Materials-screening models are commonly selected through random in-distribution evaluations, although their errors and reported uncertainties can change beyond the training domain. We present a dynamic evaluation framework that tracks predictive robustness and uncertainty reliability along a controlled distribution-shift pathway. Three Materials Project regression tasks spanning thermodynamic, electronic, and mechanical properties are partitioned by unit-cell site count into an in-distribution region and progressively shifted evaluation regimes. Five uncertainty-aware models are trained only on in-distribution data, calibrated on an independent in-distribution subset, and assessed under shift without refitting. All 45 shifted task-model-regime combinations had mean absolute error penalties above unity relative to full-database five-fold cross-validation, spanning 1.18-4.41. Although the deep ensemble achieved the lowest in-distribution test error in all three tasks, NGBoost achieved the lowest error under moderate and severe shift in every task. Post-hoc calibration reduced the mean in-distribution test calibration deviation in 13 of 15 task-model combinations, yet 14 of 15 had a larger deviation under severe shift than on the corresponding in-distribution test set. After calibration, the relative error-growth factor exceeded the uncertainty-growth factor by more than 10% in 32 of 45 shifted means. Calibration and error ranking also evolved independently, confirming that in-distribution accuracy, calibration transfer, proportional uncertainty response, and ranking utility are distinct deployment properties. Joint evaluation under an explicit shift protocol therefore provides a more informative basis for model selection than static in-distribution benchmarking alone.