Task-agnostic Prompt Compression with Context-aware Sentence Embedding and Reward-guided Task Descriptor

TMLR Paper5875 Authors

Abstract: The rise of Large Language Models (LLMs) has led to significant interest in prompt compression, a technique aimed at reducing the length of input prompts while preserving critical information. However, the prominent approaches in prompt compression often require explicit questions or handcrafted templates for compression, limiting their generalizability. We propose Task-agnostic Prompt Compression (TPC), a novel framework that generalizes compression across tasks and domains without requiring input questions or templates. TPC generates a context-relevant task description using a task descriptor trained on a curated dataset of context and query pairs, and fine-tuned via reinforcement learning with a reward function designed to capture the most relevant information. The task descriptor is then utilized to compute the relevance of each sentence in the prompt to generate the compressed prompt. We introduce 3 model sizes (Base, Large, and Huge), where the largest model outperforms the existing state-of-the-art methods on LongBench and ZeroSCROLLS, and our smallest model performs comparable to the existing solutions while being considerably smaller.

Submission Length: Long submission (more than 12 pages of main content)

Previous TMLR Submission Url: https://openreview.net/forum?id=fLvspTsIKc

Changes Since Last Submission: In accordance with AE' requests, we have adjusted the references section. We also added the link to the repository where we'll put the final code implementation.

Code: https://github.com/bliskavets/TPC

Assigned Action Editor: ~Yoshitomo_Matsubara1

Submission Number: 5875