Does Lower Precision Always Save Energy? Benchmarking Energy–Performance Trade-Offs in Weight-Quantized Large Language Model Inference on a GPU

Abstract

Large language model (LLM) inference is an increasing component of AI energy demand, while the energy effects of weight quantization depend on workload phase and implementation. We benchmarked four LLM families—Gemma 4 E4B IT, Qwen3.5 4B, Nemotron 3 Nano 4B, and Ministral 3 3B Instruct—across Q4_K_M, Q6_K, Q8_0, and BF16 weight representations on a single NVIDIA GeForce RTX 4090 using llama.cpp/CUDA. Five workloads separated prefill from autoregressive generation, and GPU energy was estimated from NVML telemetry. In PP-512, no statistically significant differences were detected under the employed measurement protocol; in PP-1024, precision affected total energy and mean GPU power. In all generation scenarios, model family, precision, and their interaction significantly affected the primary endpoints. Relative to BF16, Q4_K_M reduced dynamic energy per token by 63.6–85.7% and increased generation throughput by 85.2–160.2% across all four model families. Ministral 3 3B Instruct–Q4_K_M ranked first in TOPSIS, whereas Nemotron 3 Nano 4B–Q4_K_M required less GPU memory. Findings are limited to single-request execution on one RTX 4090/llama.cpp-CUDA stack; output quality was not assessed, and 50 ms NVML sampling limits interpretation of short prefill measurements.