Out of Time: Calibration and Stability of Tabular Foundation Models on Credit Default Vintages

Abstract

We evaluate TabPFN-3 and TabICLv2, at the defaults of tabpfn 8.5.0 and tabicl 2.1.1, out of time on Lending Club and Freddie Mac against a weight-of-evidence scorecard and a tuned gradient-boosting model, scoring each build on every later origination cohort under kill criteria written before any foundation model was fitted, with every later rule dated. At the shipped softmax temperature of 0.9 both foundation models predict too few defaults: on Lending Club observed over expected averages 1.845 for TabPFN and 1.594 for TabICL over cells, against 1.283 for gradient boosting. There the shortfall is already present on the 50,000 rows each model was conditioned on and gone at temperature one, and one intercept shift fitted on those rows removes the excess out of time; on Freddie Mac every model's typical cohort miss survives the shift. The total Brier score, within a thousandth across models, does not show the size of the gap. On Lending Club two choices a vintage study usually leaves unstated each reverse a verdict: one intercept per build or per cohort when age is specified, and the conditioning rows or a later cohort as the reference of a population stability index. TabPFN-3.5, the default of later releases, scored on one arm under one bootstrap seed, misses the default rate by less than TabPFN-3 but loses AUC with age faster. The protocol, a claim ledger that ties every number to a recorded run, and a population-stability package are released.