Panagiotis Gkilis

The Loss Curve Is Not a Sufficient Statistic: Silent Objective Failures from Sentinel-Class Collisions in Neural Codec Language Models

Masked cross-entropy is the standard training objective for sequence models. Positions that are not real targets are excluded by comparing each target against a sentinel value, conventionally ignore_index = -100 in PyTorch. We show that when this sentinel is also a valid class in...

The Loss Curve Is Not a Sufficient Statistic: Silent Objective Failures from Sentinel-Class Collisions in Neural Codec Language Models

Masked cross-entropy is the standard training objective for sequence models. Positions that are not real targets are excluded by comparing each target against a sentinel value, conventionally ignore_index = -100 in PyTorch. We show that when this sentinel is also a valid class in...