DeCAL Tokenwise Compression

Sameer Panwar

Keywords: transformer, representation, compression, encoder-decoder, context, summarization, question-answering, retrieval, nlp

TL;DR: DeCAL is a new method for tokenwise compression, focused on maximizing compression quality instead of saving compute

Abstract: This paper introduces DeCAL, a new method for tokenwise compression. DeCAL uses a native encoder-decoder language model pretrained with denoising to learn to produce high-quality compressed representations from the encoder. DeCAL applies small modifications to the encoder, with the emphasis on maximizing compression quality, even at the expense of compute. We show that DeCAL at 2x compression can match uncompressed, with usually only a minor dropoff in metrics up to 8x compression, among question-answering, summarization, and multi-vector retrieval tasks. DeCAL distinguishes itself from prior works by supporting larger chunks and significantly more compressed output per chunk.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 15691