BoA: Attention-aware Post-training Quantization without Backpropagation

Junhan Kim, Ho-young Kim, Eulrang Cho, Chungman Lee, Joonyoung Kim, Yongkweon Jeon

TL;DR: We propose a novel backpropagation-free post-training quantization algorithm that optimizes integer weights by considering inter-layer dependencies inside the attention module.

Abstract:

Post-training quantization (PTQ) is a promising solution for deploying large language models (LLMs) on resource-constrained devices.
Early methods developed for small-scale networks, such as ResNet, rely on gradient-based optimization, which becomes impractical for hyper-scale LLMs with billions of parameters.
While recently proposed backpropagation-free or transformation-based methods alleviate this issue, they ignore inter-layer interactions or use the naive nearest-rounding-based quantized weight assignment to save the heavy computational cost of weight optimization.
In this paper, we introduce a novel backpropagation-free PTQ algorithm that optimizes quantized weights by considering inter-layer dependencies.
The key innovation is the development of attention-aware Hessian matrices that capture inter-layer interactions within the attention module.
Extensive experiments demonstrate that our approach not only outperforms existing weight quantization methods but also shows good synergy with conventional methods to suppress activation outliers, leading to state-of-the-art weight-activation quantization performance.
The code will be available at https://github.com/SamsungLabs/BoA.

Lay Summary:

Post-training quantization (PTQ) is a promising solution for deploying large language models (LLMs) on resource-constrained devices.
Early methods developed for small-scale networks, such as ResNet, rely on gradient-based optimization, which becomes impractical for hyper-scale LLMs with billions of parameters.
While recently proposed methods alleviate this issue, their performance remains limited because they ignore inter-layer interactions or use the naive nearest-rounding-based quantized weight assignment to save the heavy computational cost of weight optimization.
In this paper, we introduce a novel backpropagation-free quantization algorithm that optimizes quantized weights by considering inter-layer dependencies.
The key innovation is the development of attention-aware Hessian matrices that capture inter-layer interactions within the attention module.
Extensive experiments demonstrate that our approach not only outperforms existing weight quantization methods but also shows good synergy with conventional methods to suppress activation outliers, leading to state-of-the-art weight-activation quantization performance.
Our breakthrough is particularly impactful for on-device AI, allowing real-time model inference on mobile and edge devices without the need for server access, paving the way for scalable, decentralized AI solutions.

Primary Area: Deep Learning->Everything Else

Keywords: Quantization, Hyper-scale LLMs, Attention, Hessian

Submission Number: 5733