Counteracting the overestimation of effect size in mass-univariate brain-wide association studies using cross-validation

Abstract

Abstract Background Brain-wide association effects are commonly overestimated in psychiatric neuroimaging research, leading to low replicability and generalizability of findings, and ultimately to poor translation to clinical utility. Cross-validation is currently not standardly used in univariate neuroimaging analyses but could pose a potential counteracting measure for this problem. Methods We systematically evaluated the benefit of a cross-validation procedure to expose inflated effect size estimates in mass-univariate voxel-based analysis, using body mass index (BMI) as an example predictor for gray matter. A total of n = 3401 MRI scans from a psychiatric cohort were used to investigate the impact of applying cross-validation on effect size estimates across a range of sample sizes and across different localizations with a range of underlying pooled effect sizes. Results BMI was negatively associated with gray matter density with a maximum underlying effect size of R ² p = 0.036 (in the full sample of n = 3401). However, without cross-validation, effects were substantially overestimated in small samples, with effect sizes ranging up to R ² p = 0.535. Even with an underlying null-effect, effect size estimates up to R ² p = 0.429 were reached in samples with n = 25. Applying cross-validation revealed that these inflated effects in small samples did not generalize to novel data. A minimum sample size of n = 100 was required for largest brain-wide effects to start generalizing, while samples with n = 400 were needed for smaller effects to generalize. This ‘point of initial generalizability’ was not reached for underlying null effects, even with the maximum available sample of n = 3401. Conclusions Results suggest that cross-validation is an effective measure to expose overestimated effects in small samples and thus could help to counteract low replicability and generalizability in brain-wide mass-univariate association studies. Open-source code is provided for practical implementation.