Recovery-on-the-line: Linear trends in post-quantization performance recovery
Track: tiny / short paper (up to 4 pages)
Keywords: quantization, evaluation
TL;DR: We conduct a suite of evaluations to analyze the relationship in recovery of performance metrics across several tasks in compressed LLMs.
Abstract: Many state-of-the-art large language models exceed tens of billions of parameters. To compress these models, several prior works have proposed quantizing the weights and activations of these models, using techniques like GPTQ and QuIP. An important reference for quantized language models is their ability to \textit{recover} performance metrics such as accuracy after quantization; that is, the quantized language models should be as accurate as the original base models. It remains, however, unclear, which evaluations are most needed to assess recovery. If, for instance, recovery across all tasks is strongly linear (i.e. recovery on task A is a linear function of recovery on task B), then there should exist a dense subset of (independent) evaluations that are most necessary. In this paper, we examine the trends in recovery across pairs of tasks and metrics. Drawing from prior works which have shown that in-distribution and out-of-distribution accuracy often exhibit a strong linear relationship, we show that this relationship holds for recovery of accuracy as well.
Anonymization: This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs.
Presenter: ~Shashata_Sawmya1
Format: Yes, the presenting author will definitely attend in person because they attending ICLR for other complementary reasons.
Funding: Yes, the presenting author of this submission falls under ICLR’s funding aims, and funding would significantly impact their ability to attend the workshop in person.
Submission Number: 6
Loading