Leakage-Free Evaluation of Single-Cell Treatment Response Classifiers Reveals Systematic Underpowering in Small Public Cohorts

Abstract

Background: Machine learning frameworks using single-cell RNA sequencing data are increasingly applied to predict treatment response and identify biomarker signatures. However, a pervasive methodological flaw, performing highly variable gene selection and feature scaling globally across all cells prior to cross-validation splits, introduces systematic data leakage. In clinical cohorts with small sample sizes, this leakage artificially inflates model performance, leading to over-optimistic results that fail to generalize and presenting a significant challenge to reproducible marker discovery. Methods: We applied a leakage-free nested Leave-One-Out Cross-Validation framework to two independent single-cell datasets: an inflammatory bowel disease anti-TNF cohort (n=36) and a melanoma immune checkpoint inhibitor cohort (n=18 baseline patients). Feature selection using ANOVA (k=5) and scaling were embedded strictly within each training fold. Statistical significance was evaluated using a 500-iteration nested permutation test. External bulk validation was performed on baseline melanoma patients from GSE91061 (n=33). Results: Both classifiers achieved approximately 72% nominal accuracy with moderate area under the curve values (IBD: 0.766, melanoma: 0.741). After leak correction, neither model survived nested permutation testing (IBD p=0.0878, melanoma p=0.2056). Consensus signatures emerged, featuring TNK1 (100% of IBD folds) and TBC1D10B (100%) and TNRC6B (94.4%) in melanoma. External bulk validation confirmed that no individual signature gene reached significance (0/6, p < 0.05); 83.3% showed the expected direction, though not significantly (two-sided binomial p = 0.219). Conclusions: Current public single-cell treatment response cohorts are systematically underpowered. Global preprocessing before cross-validation splits inflates significance, making properly nested permutation architectures mandatory for biomarker validation. The most stable biological signals identified in this study, notably TNK1, warrant targeted validation in larger prospective patient cohorts.