Frozen Brain-MRI Foundation Models Are Site Fingerprints

Abstract

Frozen foundation-model (FM) embeddings are increasingly used as off-the-shelf brain-MRI representations, on theassumption that they capture anatomy. We audit what they actually encode and find that acquisition site is a large,intrinsic component of the representation. Across two independent cohorts (ABIDE-I, ABIDE-II), three frozen 3-Dencoders (brain-pretrained, CT-pretrained, and randomly initialized), and every network depth, site is linearly decodableat roughly 0.9 balanced accuracy at deep layers, exceeding the decodability of every clinical or demographic variable (sex,age, autism diagnosis) at every layer. The effect is intrinsic rather than learned: a randomly initialized encoder is alreadya ∼0.9 site classifier on both cohorts and across three architecture families (Swin, ViT, ResNet), and site is decodable at∼0.95 directly from the raw downsampled image with no encoder, so the fingerprint reflects low-level image statisticsthat any encoder preserves rather than a product of pretraining. Residualizing measured population covariates leavessite decodability essentially unchanged, indicating an acquisition- rather than population-driven effect. A nonlinearprobe matches the linear one, so the fingerprint is fully linearly accessible. The site subspace is removable post hocby iterative null-space projection or ComBat (site decodability 0.94 → 0.07/0.00), and is a site-attribution concern forshared or federated embeddings; but for dense segmentation this removal is not free, because site and anatomy occupyan entangled linear subspace (a matched-rank random-direction projection is Dice-neutral, whereas removing the sitesubspace is destructive). We recommend site-audited use of frozen brain-MRI FMs and release an open audit toolkit.