LLM-QAT: Data-Free Quantization Aware Training for Large Language Models

Anonymous

Abstract: Several post-training quantization methods have been applied to large language models (LLMs), and have been shown to perform well down to 8-bits. We find that these methods break down at lower bit precision, and investigate quantization aware training for LLMs (LLM-QAT) to push quantization levels even further. We propose a data-free distillation method that leverages generations produced by the pre-trained model, which better preserves the original output distribution and allows quantizing any generative model independent of its training data, similar to post-training quantization methods. In addition to quantizing weights and activations, we also quantize the KV cache, which is critical for increasing throughput and support long sequence dependencies at current model sizes. We experiment with LLaMA models of sizes 7B, 13B, and 30B, at quantization levels down to 4-bits. We observe large improvements over training-free methods, especially in the low-bit settings.

Paper Type: long

Research Area: NLP Applications

Contribution Types: Approaches to low-resource settings

Languages Studied: English

Preprint Status: There is a non-anonymous preprint (URL specified in the next question).

A1: yes

A1 Elaboration For Yes Or No: Section 5 Conclusion and Limitations

A2: yes

A2 Elaboration For Yes Or No: A.7 Broader Impact

A3: yes

B: no

B1: n/a

B2: n/a

B3: n/a

B4: n/a

B5: n/a

B6: yes

C: yes

C1: yes

C2: yes

C3: yes

C4: yes

D: no

E: no