Position IDs Matter: An Enhanced Position Layout for Efficient Context Compression in Large Language Models

ACL ARR 2025 May Submission5674 Authors

Abstract: Using special tokens (e.g., gist, memory, or compressed tokens) to compress context information is a common practice for large language models (LLMs). However, existing approaches often neglect that position encodings inherently induce local inductive biases in models, causing the compression process to ignore holistic contextual dependencies. We propose **Enhanced Position Layout (EPL)**, a simple yet effective method that improves the context compression capability of LLMs by only adjusting position IDs, the numerical identifiers that specify token positions. EPL minimizes the distance between context tokens and their corresponding special tokens and at the same time maintains the sequence order in position IDs between context tokens, special tokens, and the subsequent tokens. Integrating EPL into our best performing context compression model results in 1.9 ROUGE-1 F1 improvement on out-of-domain question answering datasets in average. When extended to multimodal scenarios, EPL brings an average accuracy gain of 2.6 to vision compression LLMs.

Paper Type: Long

Research Area: Language Modeling

Research Area Keywords: LLM/AI agents, fine-tuning, continual learning, prompting, sparse models

Contribution Types: Model analysis & interpretability, Reproduction study, Approaches low compute settings-efficiency, Theory

Languages Studied: English

Keywords: LLM/AI agents, fine-tuning, continual learning, prompting, sparse models

Submission Number: 5674