LOAD: Enabling Low-Bit Group Quantization for Vision Transformers via Adaptive Data Formats

Abstract

Vision Transformers (ViTs) achieve state-of-the-art performance across various vision tasks but pose significant deployment challenges due to their high computational complexity and memory footprint. Post-training quantization (PTQ) offers an efficient, retraining-free compression strategy; however, existing solutions typically enforce a uniform quantization scheme (fixed bit-width and data format) across all layers, constraining the potential accuracy-efficiency trade-off. We propose LOAD-ViT , a novel PTQ framework that leverages both mixed-precision and mixed-data-format quantization to allocate resources at a fine granularity based on layer sensitivity. LOAD-ViT optimizes per-layer configurations via a calibration-based sensitivity metric and a constrained search algorithm, minimizing accuracy loss under strict hardware budgets. To support this scheme, we design a dedicated hardware accelerator featuring heterogeneous compute engines and lightweight format-conversion units, facilitating the efficient execution of mixed formats. Extensive evaluations on various ViT models and benchmarks demonstrate that LOAD-ViT achieves up to 5.3 × speedup and 4.2 × energy savings compared to state-of-the-art quantization accelerators, with negligible accuracy degradation.