Auditing Evaluation Leakage in Formula 1 Race-Strategy Machine Learning: A Track-Identity Confound and a Corrected Protocol

Abstract

Machine learning models for Formula 1 race strategy are typically evaluated with random train/test splits over lap-level data, which lets adjacent laps from the same race leak between train and test. We quantify this directly on lap time and pit-stop prediction using 2023 FastF1 telemetry (22 races, 20,447 laps). Removing this leakage collapses a naively evaluated XGBoost lap-time model's R² from 0.994 to −2.26 on unseen circuits once track-identity features are also removed, showing that absolute lap time is dominated by track identity rather than the race-condition features (tyre life, stint, weather) typically used to justify these models. A track-aware evaluation (holding out drivers rather than circuits) restores an honest R²=0.991 (MAE=0.54s, vs. a persistence baseline's R²=0.949, MAE=0.61s). The same corrected pipeline yields a pit-stop-next-lap classifier (F1=0.303, ROC-AUC=0.888, vs. a logistic regression baseline's ROC-AUC=0.669) that trails current published deep-learning approaches on a closely related task (Sasikumar et al. 2025: F1=0.81), which we report as a limitation rather than a contribution. Applying the lap-time model without retraining to 2024 data shows a substantial generalization gap (R² 0.991 to 0.172; pit-stop ROC-AUC 0.888 to 0.721), suggesting season-over-season performance changes are not captured by the current feature set. We release code, corrected metrics, and a leakage decomposition to support more reproducible evaluation in this domain.