Each Tree Corrects the Last: What Gradient Boosting Actually Learns

Abstract

Gradient boosting is commonly described as a sequence of weak trees in which each new learner corrects errors left by the current ensemble. This description is computationally accurate but scientifically incomplete. Every correction is conditioned on a declared loss function, an initial model, a learning rate, a stage budget, a base-learner class, and the development data from which the residual structure is derived. This paper introduces the Boosting Validity Map, an eight-level framework for separating the initial model, correction target, base learner, step size, stopping rule, temporal transport, refit stability, and predictive interpretation. Unlike the seven-level maps used for static estimators earlier in the series, this map separates the initial model from the correction target because every subsequent learner is defined conditionally on the current ensemble state. The empirical study uses 392 complete observations from the UCI Auto MPG dataset. Model years 1970–1979, comprising 307 vehicles, form the development period. Within development, observations from 1970–1976 are used for training and those from 1977–1979 for internal validation. Model years 1980–1982, comprising 85 vehicles, remain frozen as a temporally separated benchmark. Internal model selection evaluates twelve configurations formed by crossing learning rates 0.01, 0.03, 0.05, and 0.10 with maximum tree depths 1, 2, and 3. Every candidate uses 300 boosting stages and squared-error loss. Selection minimizes internal validation MAE and then RMSE without consulting the final temporal benchmark. The selected configuration uses a learning rate of 0.1, depth-1 regression trees, and 300 stages. On the temporal benchmark, it records an MAE of 3.945 mpg, an RMSE of 5.154 mpg, an R² of 0.263, and a mean prediction bias of −3.211 mpg. The negative bias indicates systematic underprediction under the convention predicted minus observed. The complete boosting trajectory illustrates the distinction between optimization and temporal validity. Development MAE declines from 4.971 mpg after the first stage to 1.232 mpg after 500 stages. Temporal MAE falls from 10.655 to 3.836 mpg over the same interval, but the temporal improvement is substantially slower and non-monotonic. The lowest observed temporal MAE in the recorded trajectory occurs retrospectively at stage 485, where it reaches 3.833 mpg. This stage is reported only as a descriptive benchmark audit. Selecting it after observing the 85 temporal outcomes would convert the frozen benchmark into validation data and constitute information leakage. The prespecified 300-stage configuration therefore remains the valid selected model. A separate post-selection shrinkage audit examines learning rates from 0.01 through 0.20 across different stage budgets. Learning rate 0.20 is added solely as a diagnostic high-step-size trajectory. It is not one of the twelve internal model-selection candidates and cannot alter the selected configuration. The audit demonstrates that learning rate and stage count define a joint optimization trajectory and should not be interpreted independently. The paper also compares squared-error, Huber, and absolute-error correction paths. These losses produce different correction targets and temporal behavior. Their benchmark results are treated as diagnostic analyses rather than as retrospective alternatives to the prespecified final model. Stability is evaluated through 200 complete development-bootstrap refits. The median standard deviation of benchmark predictions across refits is 0.833 mpg, while the 90th percentile is 1.174 mpg. This fitted-model sensitivity is smaller than the persistent temporal bias of −3.211 mpg. The result demonstrates that algorithmic repeatability does not eliminate a directional error shared across fitted trajectories. Under the same fixed temporal protocol, the selected Gradient Boosting model improves upon the Random Forest examined in Paper 10, whose temporal MAE is 4.133 mpg, but it does not outperform the standalone decision tree from Paper 09, whose temporal MAE is 3.419 mpg. This ranking is specific to the dataset, chronology, selection rule, metrics, and implementations used in the series; it does not establish a universal hierarchy among model families. The central conclusion is that each tree can correct the last prediction, but no tree can correct an information boundary it has never observed. The accompanying reproducibility package contains the UCI Auto MPG data snapshot, executable source code, complete internal-selection results, stage trajectories, shrinkage and loss audits, 200 bootstrap refits, importance analyses, source-generated figures, dependency specifications, verification materials, a manifest, and SHA-256 checksums.