Variance-Adaptive Self-Regularizing Ensemble Learning for Robust Predictions in Small-Data Regimes

Abstract

The small-data regime remains one of the most important and pressing problems in machine learning. With only a few hundred sample points in the training set, conventional ensemble methods are potentially unreliable because they have high variance, low generalization and are sensitive to individual points in the dataset, which can have a negative impact on any consumer application or safety-critical application. These problems are addressed by current regularization approaches one at a time, but there is no single approach that flexibly tunes the regularization level according to the statistics of the ensemble during training. This paper presents SelfReg-Ensemble, a variance-adaptive self-regularizing ensemble learning framework which tracks the variance at each training step of all base learners and dynamically adjusts the regularization intensity to keep the variance of the ensemble predictions at an acceptable level. The framework consists of three complementary components: (i) Variance Monitoring Module (VMM) to monitor the variance of predictions performed by the ensemble of members, (ii) Self-Regularization Controller (SRC) to adaptively map the observed variance to a regularization coefficient, using a sigmoid-bounded adaptive learning schedule, and (iii) Diversity-Preserving Aggregation Layer (DPAL) based on a weighted stacking with an entropy-regularized softmax voting mechanism, to avoid ensemble collapse. We offer rigorous theoretical analysis of the proposed framework that guarantees convergence and provides bounds on variance. These guarantees are formally established for convex, gradient-based learners; for the tree-based learners used in our experiments they serve as qualitative guidance and are supported empirically rather than formally proved. SelfReg-Ensemble is tested on 12 benchmark datasets from medical, financial and IoT domains, each with a total of fewer than 500 samples (N denotes total dataset size; effective per-fold training sizes Ntr range from 52 to 432 samples after stratified 90/10 splitting) and consistently outperforms ten state-of-the-art baselines, with on average 6.8% higher AUROC than XGBoost v2.0.3 and 5.2% higher than the actual strongest average baseline, Sub-Network Ensemble (85.6% average AUROC), 9.3% lower prediction variance, and 4.1% higher F1-score. The proposed framework is lightweight, modular and easily deployable in resource-limited consumer electronics environments.