What Does MLX 4-bit Cost? A Controlled Audit of Quantization for Code Generation on Apple Silicon

Abstract

Controlled measurement of what MLX default 4-bit quantization costs five code models on HumanEval+ and MBPP+ against bf16 baselines converted from the same weights. Pooled: 220 discordant pairs favour bf16 against 125 (McNemar p < 1e-6); the loss shrinks with model size at +0.52 points per billion parameters. A calibrated AWQ arm does not recover it. Also documents why differencing against a vendor's published figure is unsound, and five harness defects that scored correct answers as failures.