Context Compression for Auto-regressive Transformers with Sentinel Tokens
Submission Type: Regular Short Paper
Submission Track: Efficient Methods for NLP
Submission Track 2: Language Modeling and Analysis of Language Models
Keywords: context compression, key-value cache compression
TL;DR: context compression
Abstract: The quadratic complexity of the attention module makes it gradually become the bulk of compute in Transformer-based LLMs during generation. Moreover, the excessive key-value cache that arises when dealing with long inputs also brings severe issues on memory footprint and inference latency. In this work, we propose a plug-and-play approach that is able to incrementally compress the intermediate activation of a specified span of tokens into compact ones, thereby reducing both memory and computational cost when processing subsequent context.
Experiments on both in-domain language modeling and zero-shot open-ended document generation demonstrate the advantage of our approach over sparse attention baselines in terms of fluency, n-gram matching, and semantic similarity. At last, we comprehensively profile the benefit of context compression on improving the system throughout. Code is available at \url{https://github.com/DRSY/KV_Compression}.
Submission Number: 1375
Loading