The Loss Curve Is Not a Sufficient Statistic: Silent Objective Failures from Sentinel-Class Collisions in Neural Codec Language Models

Abstract

Masked cross-entropy is the standard training objective for sequence models. Positions that are not real targets are excluded by comparing each target against a sentinel value, conventionally ignore_index = -100 in PyTorch. We show that when this sentinel is also a valid class index of the model's output layer, the corresponding class is removed from the objective entirely: every target instance of it is discarded before the loss is computed, while all other classes train normally. The reported loss remains a correct measurement of a corrupted objective, so its curve is indistinguishable from that of a healthy run, and no curve-based diagnostic can detect the fault. We document this in a 730M-parameter autoregressive/non-autoregressive neural codec language model for multilingual text-to-speech, in which the end-of-sequence identifier and the loss sentinel were the same integer. The model trained for fifteen epochs on a monotonically descending loss curve and never learned to terminate generation. We contribute (i) a formal statement of the collision condition and its gradient consequence; (ii) a controlled 40-line reproduction in which the faulty and corrected arms reach final losses of 0.0035 and 0.0034 while differing by six orders of magnitude in stop-token probability; (iii) held-out measurements from the full-scale model before and after correction; (iv) an independent finding that training loss and task capability decouple after competence, such that lowest-loss checkpoint selection returns a materially inferior model; and (v) two deterministic, released detectors that decide the fault at initialisation from the output width and one integer.