The Path to the Minimum Matters: What an Optimizer Actually Changes

Abstract

An optimizer does not ordinarily redefine the statistical target encoded by a loss function, but it determines which solution is reached, how closely it is approached, how stable that approach is, and how much computational work is required. Therefore, a valid objective does not by itself guarantee a valid optimization process. This paper introduces the Optimization Validity Map, a framework connecting objective specification, gradient correctness, numerical geometry, convergence evidence, initialization stability, computational accounting, predictive agreement, and transport. It extends the distinction between loss validity—whether the learning objective represents the intended target—and optimization validity—whether the numerical procedure approaches that objective under a defensible configuration and budget. The framework is evaluated on 392 vehicles from the UCI Auto MPG dataset. Model years 1970–1979, comprising 307 vehicles, form the development period, while model years 1980–1982, comprising 85 vehicles, serve as a temporally separated final benchmark. The target, 29-term pairwise feature representation, preprocessing pipeline, Ridge mean-squared-error objective, regularization strength, and temporal evaluation protocol remain fixed. Because the resulting problem is convex and admits a closed-form Ridge solution, optimization error can be measured against an exact reference rather than inferred only from software termination flags. Six prespecified optimization configurations are evaluated from 40 shared initializations, producing 240 principal runs. Their behavior is assessed using objective gap, gradient norm, parameter distance, predictive distance, temporal mean absolute error, initialization sensitivity, and observed computational work. Under a nominal limit of 300 data passes, L-BFGS-B reaches a median objective gap of approximately 3.18 × 10⁻¹². Heavy-ball momentum and Nesterov acceleration reach median gaps of approximately 8.89 × 10⁻⁴ and 1.11 × 10⁻³, respectively. The prespecified stochastic gradient descent, gradient descent, and Adam configurations remain substantially farther from the exact solution under the same nominal accounting rule. The exact Ridge solution records a temporal benchmark MAE of 3.794 mpg. L-BFGS-B and the two momentum-based configurations approach both its numerical solution and predictive behavior, while the remaining prespecified configurations retain larger numerical and predictive discrepancies at the end of the nominal budget. A separate computational-work audit shows that nominal passes are not equivalent to identical computational cost. L-BFGS-B uses a median of 130.5 accepted iterations and 147 joint objective-gradient evaluations. Full-gradient methods use 300 vectorized gradient evaluations, while stochastic gradient descent performs 92,100 individual-observation gradients per run. Wall-clock time is reported only descriptively because it depends on implementation, vectorization, software, and hardware. Adam and stochastic gradient descent are evaluated here as fixed, budget-constrained configurations in a deterministic, low-dimensional, full-objective setting. This lies outside the large-scale, stochastic, mini-batch, and non-convex regimes in which their practical advantages commonly arise. Consequently, their results characterize these configurations on this problem and do not establish a universal ranking of optimizer families. The findings show that convergence must be treated as an evidence claim rather than as a software status or optimizer name. Numerical proximity to the optimum and predictive adequacy are related but distinct claims, and nominal iteration counts alone do not establish equal computational work. The record includes a complete reproducibility package containing source code, data, experimental protocols, exact-reference calculations, optimizer configurations, results from all shared initializations, computational-work measurements, predictions, figures, structured outputs, integrity manifests, and SHA-256 checksums.