ConQuist: Condition Number Aware Quantization for LLMs

Ritik Mishra, M. Tanveer

Keywords: Post-Training Quantization, Mix-Precision Quantization, Condition Number

TL;DR: ConQuist presents a novel post-training quantization method that introduces the condition number as a metric to quantify layer sensitivity in LLM quantization.

Abstract: Post-training quantization (PTQ) of large language models (LLMs) has emerged as a promising technique in reducing the computational cost at inference time. Uniformly quantizing all weights and activations to 4-bit significantly degrades performance, due to the high quantization error caused by outliers present in activations. To mitigate this issue, we propose ConQuist, a PTQ method leveraging mixed precision quantization based on the condition number of each layer. The condition number quantifies the sensitivity of a layer’s output to small perturbations in its activations; hence, layers exhibiting high condition numbers are prone to high quantization error. ConQuist identifies layers with higher condition numbers and allocates them higher precision (e.g., 5-bit), while quantizing the rest to 4-bit. We also provide a theoretical foundation that relates activation sensitivity to the condition number. Furthermore, we have empirically shown that our proposed ConQuist outperforms uniform PTQ methods, achieving up to 20\% lower perplexity on a variety of benchmarks.

Primary Area: optimization

Submission Number: 23809