KVLinC: KV Cache Quantization with Hadamard Rotation and Linear Correction

ICLR 2026 Conference Submission18633 Authors

Keywords: KV cache quantization, KV cache compession, Efficient Inference, Large Language Models

TL;DR: We propose a KV cache quantization technique that leverages Hadamard transformation and trainable modules to minimize error in attention.

Abstract: Quantizing the key-value (KV) cache is a promising strategy for improving the inference efficiency of large language models (LLMs). However, aggressive quantization to very low precision (e.g., 2 bits) introduces significant errors in the stored key and value tensors, which propagate through the dot-product attention mechanism and ultimately degrade generation quality. To address this, we propose KVLinC, a framework to mitigate attention errors introduced by KV cache quantization in the extreme low-precision regime. KVLinC combines a Hadamard rotation, which reduces quantization error in values, with lightweight linear correction adapters that explicitly compensate for errors introduced by quantized keys. Across extensive evaluations on the LLaMA, Qwen2.5, and Qwen3 model families, KVLinC consistently matches or surpasses strong baselines while achieving higher KV-cache compression. Furthermore, we implement a custom attention kernel that results in upto 2.55x faster inference compared to Flash Attention baseline, enabling efficient long-context LLM inference.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 18633