Risk-Based Software Test Prioritization via Machine-Learning Defect Prediction: Label-Feature Circularity, Clean Baselines, and an Honest Empirical Study on Five Open-Source Repositories

Abstract

Risk-based software testing seeks to allocate CI test-execution effort proportional to estimated defect likelihood, typically via a machine-learning classifier trained on repository process metrics. We show that the standard formulation harbours a fatal label-feature circularity: when the label is defect_prone = 1 iff bug_fix_commits >= 1 and bug_fix_commits is simultaneously used as an input feature, the classifier learns to read its own label. On a corpus of 296,457 file instances from five mature open-source repositories (Elasticsearch, Spring Boot, Hadoop, Kafka, Express), an honesty test confirms the failure mode: ranking files by bug_fix_commits alone gives 55.52% top-10% coverage and perfect AUC, inflating naive ML numbers by roughly ten percentage points. We propose a clean protocol that (i) drops the two keyword-derived leaky features (bug_fix_commits, commit_frequency), (ii) introduces non-trivial single-feature baselines (rank-by-commit_count, rank-by-code_churn), and (iii) evaluates on a source-only held-out test fold. Under this protocol, Random Forest on the five remaining process metrics achieves 46.47% top-10% coverage (AUC 0.9143, Brier 0.0772), Gradient Boosting 45.29% (AUC 0.9036, Brier 0.0822), versus the best single-feature baseline (rank by commit_count) at 43.17%. Aggregating over 10 stratified splits, the ML lift over the strongest baseline is +3.64 percentage points (95% CI [+3.32, +3.95], paired t-test p = 8.5e-10, RF wins 10/10 seeds). For deployment Gradient Boosting is Pareto-superior on latency (104 us/row vs 12 ms for RF, 115x faster) with comparable calibration. We disclose an honest negative finding: even the clean feature set does not generalize temporally (time-aware AUC 0.5246, top-10% coverage 12.44%); the paper's random-split numbers should be read as defect-distribution characterisation, not forward prediction. Preprint status: Under peer review at Journal of Systems and Software (JSS, Elsevier) and PeerJ Computer Science. Not peer-reviewed on Zenodo. Full reproducibility bundle (LaTeX source, figures, results CSVs, honesty test scripts) included as supplementary ZIP.