ATAC: Abstractive Token-Level Question-Agnostic Prompt Compression

ICLR 2026 Conference Submission21813 Authors

Keywords: LLMs, prompt compression, Question-Agnostic Prompt Compression, deep learning, NLP

Abstract: Motivated by the high costs of using black-box Large Language Models (LLMs), we introduce a novel prompt compression paradigm, under which we use smaller LLMs to compress inputs for the larger ones. We present the first comprehensive LLM-as-a-compressor benchmark spanning $25$ open- and closed-source models, which reveals significant disparity in models' compression ability in terms of (i) preserving semantically important information (ii) following the user-provided compression rate (CR). We further improve the performance of gpt-4.1-mini, the best overall vanilla compressor, with Textgrad-based compression meta-prompt optimization. We also identify the most promising open-source vanilla LLM---Qwen3-4B---and post-train it with a combination of supervised fine-tuning (SFT) and Group Relative Policy Optimization (GRPO), pursuing the dual objective of CR adherence and maximizing the downstream task performance. We call the resulting model Cmprsr and demonstrate its superiority over both extractive and vanilla abstractive compression across the entire range of compression ratios on lengthy inputs from MeetingBank and LongBench as well as short prompts from GSM8k. The latter highlights Cmprsr's stable performance for varying input types. Moreover, Cmprsr closely follows the requested compression ratio, offering fine control over the cost-quality trade-off.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 21813