PIC: Unlocking Long-Form Text Generation Capabilities of Large Language Models via Position ID Compression

ACL ARR 2025 February Submission6348 Authors

Abstract: Long-context understanding is crucial for large language models (LLMs) and has become a fundamental capability for most LLMs. However, beyond the focus on "input-long", the ability to "output-long" is equally significant, yet it remains underexplored. To address this limitation, we propose a simple, efficient, and plug-in approach, **Position ID Compression (PIC)**, to unlock the long-form text generation potential of LLMs. The idea is straightforward: by compressing the position ids of the context, we provoke and guide LLMs to generate coherent and longer output. Specifically, we find that directly reducing the position ids by a fixed ratio significantly impacts the generation quality. To mitigate this, we propose two variants of PIC: **NTK-aware PIC** and **Dynamic PIC**. Without additional training, both methods enable LLMs to extend their generation length by approximately 1.5 times without compromising generation quality. Furthermore, by integrating supervised fine-tuning (**SFT**) with PIC, we propose **PIC-SFT**, which further improves LLMs' long-form text generation capabilities, achieving top performance on HelloBench and LongBench-Write. Extensive experiments demonstrate the effectiveness of our approach.

Paper Type: Long

Research Area: Generation

Research Area Keywords: text-to-text generation,inference methods

Contribution Types: NLP engineering experiment, Approaches low compute settings-efficiency

Languages Studied: English,Chinese

Submission Number: 6348