When SHAP explains a leaked target: a cautionary study of an XGBoost and multi-level SHAP pipeline for S&P 500 direction prediction

Abstract

Accurate prediction of stock price movements is of substantial importance to investors, portfolio managers, and financial regulators, and although machine learning models are increasingly deployed in quantitative finance, the limited transparency of many such models remains a non-trivial barrier to broader adoption in high-stakes decision-making environments, particularly where auditability and accountability are required. This study presents an end-to-end interpretable prediction pipeline that integrates multi-category technical indicator engineering, XGBoost gradient boosting classification, and comprehensive SHAP (SHapley Additive exPlanations) analysis for daily direction forecasting of the S&P 500 index, together with a rigorous diagnostic evaluation of that pipeline. A systematic feature engineering stage constructs 32 technical features organized into six categories—returns, trend indicators (moving average ratios, MACD, RSI, Bollinger Bands), multi-scale volatility, volume dynamics, momentum oscillators (ROC, ATR, Stochastic Oscillator, Williams %R, CCI), and lagged returns—which are fed into an XGBoost classifier trained on a sequential 80/20 train–test split to prevent look-ahead bias. On the hold-out set the classifier attains 63.84% accuracy and a macro-averaged F1-score of 0.6110. Contextualizing this figure, however, overturns its interpretation. The majority-class (“always-Down”) rule already attains 59.32% on the same split, and a data-provenance audit reveals that the widely used public source file contains every calendar day, with 31.1% of rows being exact forward-filled copies of the previous trading session. Because the target is defined by the strict inequality \(P^{\textrm{c}}_{t+1}>P^{\textrm{c}}_{t}\) , every session whose successor is a forward-filled copy—each Friday and each Saturday—is labelled “Down” by construction, rendering 31.0% of all labels independent of market behaviour. Consequently a one-bit rule that never observes a price (“predict Up iff the previous return is exactly zero”) reaches 63.84%, and a calendar-only rule reaches 71.19%, both matching or exceeding the classifier. Decomposing the model’s accuracy shows it scores 95.37% on rows whose label the calendar had already fixed but only 49.19% on the 246 rows carrying a genuine next-day move—below that subset’s own 58.54% majority rule. Once the forward-filled rows are removed, discriminative power collapses to chance (AUC 0.527 on a single split; 0.476 pooled across six walk-forward folds with per-fold hyper-parameter re-optimization), the model falls below the majority baseline, and a cost-aware backtest yields a Sharpe ratio of \(-0.24\) at 10 bps against \(+0.59\) for buy-and-hold. SHAP attributions are further shown to be unidentified within a volatility cluster whose pairwise correlations reach \(r=0.947\) : the members’ individual credits sum to 2.64 times the cluster’s joint contribution, and retraining on one representative per correlated group—discarding fourteen of the thirty-two features—does not degrade performance. The multi-level SHAP machinery faithfully explains this leakage rather than any market mechanism—most visibly, 108 of 354 test observations pile up at exactly zero lagged return and receive systematically positive attributions. The contribution of this work is therefore methodological rather than predictive: it provides a reproducible, auditable SHAP-on-XGBoost pipeline, and uses it to demonstrate how a routinely used public dataset combined with a strict-inequality target manufactures label leakage that post-hoc explanation methods will rationalize with apparent precision. We distil the analysis into a validation checklist for interpretable machine learning on financial tabular data.