Out of Time: A Pre-Registered Vintage, Calibration and Stability Evaluation of Tabular Foundation Models for Credit Default

Abstract

TabPFN-3 and TabICLv2 are read as shipped beside a weight-of-evidence scorecard and a tuned gradient-boosting model, out of time on Lending Club and Freddie Mac vintages, under pre-registered kill criteria. At the softmax temperature of 0.9 that tabpfn 8.5.0 and tabicl 2.1.1 ship, both models' mean probability on their own context rows is, in point, 0.656 to 0.736 of the realised default rate on every context of the Lending Club expanding arm, and at 1.0, the setting at which tabpfn 9.0.0 ships TabPFN-3.5, observed over expected holds one; an attribution of TabPFN-2.6's weaker calibration on random folds to smoothed probabilities is supported for neither model, and on Freddie Mac's expanding arm both models' own slopes at 1.0 lie below one beyond the interval, a logit more spread than the outcomes warrant. On Lending Club no discrimination, calibration or stability kill fires for TabPFN (calibration at the margin), while for TabICL the calibration kill fires at 0.9 and not at 1.0, on one build, and the stability kill fires, on the oldest builds, and the context-window hypothesis is killed for both. On Freddie Mac's registered matrix, with the loan amount as a ratio to the conforming limit, the calibration kill fires for both at 0.9 and for neither at 1.0, and the discrimination and stability kills for neither; on the nominal-dollar matrix reported beside it the stability kill fires for TabPFN and the calibration kill at 1.0 too. On Lending Club TabICL's slope of AUC on age is no worse than that of a gradient-boosting model fitted on the same 50,000 context rows with one intercept per build and worse beyond the interval with one per cohort, and pooled over every cell its PSI excess over that model inverts with the reference, at the margin with the cohorts resampled, so a vintage study has to report both identifications of age and both references. Code, recorded runs and the claim ledger: https://github.com/primaryaesthetics/out-of-time-tfm-credit. Pre-registered protocol: https://doi.org/10.5281/zenodo.22803069.