The Metric Chooses the Winner: What Model Evaluation Actually Rewards

Abstract

A model is not inherently “best” independently of the rule used to evaluate it. Evaluation metrics determine which errors receive greater weight, which properties of a prediction system become visible, and which model appears superior. Consequently, model selection is incomplete until the metric, population, uncertainty, and decision context defining success are specified. This paper introduces the Metric Validity Map, a seven-level framework connecting estimand validity, error geometry, scale and transformation behavior, aggregation, population structure, ranking uncertainty, and decision alignment. It extends the distinction developed across the preceding studies between the target defined by the loss function, the solution reached by the optimizer, and the evidence ultimately rewarded by evaluation. The framework is examined using 392 vehicles from the UCI Auto MPG dataset. Model years 1970–1979, comprising 307 vehicles, form the development period, while model years 1980–1982, comprising 85 vehicles, serve as a temporally separated final benchmark. Six fixed prediction systems produce 510 final predictions. These predictions remain unchanged throughout the principal evaluation so that only the evaluation rule varies. Eight ranking metrics are examined, including mean absolute error, root mean squared error, coefficient of determination, median absolute error, mean absolute percentage error, weighted absolute percentage error, the 90th percentile of absolute error, and maximum absolute error. Ranking stability is assessed using 10,000 paired bootstrap resamples. Additional analyses examine performance across target ranges and test the behavior of percentage-based metrics under a translation of the measurement origin. The identity of the observed winner changes with the metric. Huber regression records the lowest MAE at 3.520 mpg and the lowest observed RMSE at 4.6822 mpg. Random Forest records the lowest median absolute error at 2.592 mpg. Gradient Boosting obtains the lowest MAPE at 10.856%, while Ridge regression records the lowest 90th-percentile absolute error at 7.806 mpg and the lowest maximum absolute error at 11.429 mpg. The difference between the lowest observed RMSE values is extremely small. Although Huber obtains the minimum point estimate, Ridge wins a larger proportion of paired bootstrap resamples—40.17% compared with 25.87% for Huber. This demonstrates that an observed minimum does not necessarily establish a resolved model ranking. Evaluation by consumption range reveals performance differences and bias patterns hidden by aggregate scores. A translation stress test changes the MAPE ranking without altering any residual, demonstrating that percentage-based evaluation can depend on the arbitrary origin of the measurement scale rather than only on predictive error. The findings show that metrics do not merely report model quality. They operationalize a particular definition of quality and can change the apparent winner even when every prediction remains fixed. No individual metric provides a universally sufficient evaluation, and agreement between aggregate scores does not guarantee agreement across populations, error severities, transformations, or decisions. The record includes a complete reproducibility package containing source code, data, frozen predictions, metric definitions, ranking tables, paired bootstrap results, subgroup analyses, translation stress tests, figures, structured outputs, integrity manifests, and SHA-256 checksums.