Beyond Accuracy: Measuring Reasoning Consistency Under Quantization in Small Language Models

Abstract

Quantization enables large language models to run on consumer hardware, but evaluation has focused almost exclusively on mean accuracy, leaving output consistency largely unexamined. We address this gap by evaluating Phi-3.5-mini-instruct at three precision levels, FP16, 4-bit, and 2-bit, across 50 GSM8k questions with N = 15 repeated runs per question to measure whether compressed models produce consistent answers. We find that accuracy, consistency, and answer entropy all degrade monotonically with compression. Most notably, the correct-but-inconsistent rate, questions the model answers correctly on some runs but fails on others, rises from 42% at FP16 to 58% at 2-bit. These results suggest that mean accuracy substantially understates the reliability cost of aggressive quantization, demonstrating that consistency metrics should be reported alongside standard accuracy in edge deployment evaluations.