Dual Grained Quantization: efficient fine-grained quantization for LLM

Anonymous

Abstract: Large Language Models (LLMs) demonstrate considerable potential across a range of tasks; however, they pose significant challenges due to their extensive memory requirements and computational demands. Fine-grained quantization effectively preserves model performance during aggressive weight compression, yet its inefficiency on hardware platforms hinders its applicability in real-world production environments. To enhance hardware efficiency while preserving the performance of fine-grained quantization, we propose a novel quantization framework, Dual Grained Quantization (DGQ), employing a W4A8 configuration specifically tailored for LLMs. By employing a dual-phase search strategy, DGQ minimizes quantization error without significantly extending quantization time. To improve the accuracy of W4A8-configured LLMs, we introduce aggressive selective equalization. This approach is grounded in the observation that key weights and outliers frequently coexist within the same channels. Comprehensive experiments with our W4A8 CUDA kernel highlight DGQ's exceptional performance, delivering speedups of 1.37$\times$ and 2.5$\times$ over standard INT8 and FP16 kernels, respectively, while preserving the superior performance of fine-grained quantization.

Paper Type: long

Research Area: Efficient/Low-Resource Methods for NLP

Contribution Types: Approaches to low-resource settings

Languages Studied: English