TrustShift: Shift Type, Not Shift Magnitude, Determines Machine-Learning Failure Modes - A Cross-Domain Audit Protocol and a Failure Taxonomy

Abstract

Deployment evaluation implicitly assumes that a larger distribution shift means greater risk. Across four domains, we find this assumption false. We introduce TrustShift, a single pre-registered protocol measuring discrimination, calibration, and subgroup reliability, and apply it identically to four real deployment shifts: clinical risk prediction (NHANES to BRFSS), mental-health text classification (Kaggle to Reddit/Twitter), mortgage-approval prediction (temporal and cross-state shift over 42M U.S. loan records), and network-intrusion detection (CIC-DDoS2019 to CICIDS2017). A lending model under large measured covariate shift (domain-classifier AUC up to 0.80) degrades on no axis, while a mental-health model at comparable magnitudes loses up to 0.39 AUC: magnitude does not predict damage. The shift type does. Concept shift breaks every axis; novel-label shift blinds a detector to unseen classes while its in-domain AUC reads a perfect 1.00; covariate shift alone breaks nothing; mixed shift breaks calibration silently while discrimination holds. In-domain metrics anticipated none of this, yet three inexpensive diagnostic probes correctly distinguish the shift type before deployment. Post-hoc recalibration then cuts calibration error by one to two orders of magnitude in every domain but never restores lost discrimination or subgroup reliability. We contribute a protocol and a four-domain benchmark, and distill the result into a failure taxonomy mapping a pre-deployment diagnosis to the trustworthiness axis at risk. Preprint - not peer reviewed. Submitted for publication.