LOGO --- Long cOntext aliGnment via efficient preference Optimization

Zecheng Tang, Zechen Sun, Juntao Li, Qiaoming Zhu, Min Zhang

Keywords: Long-context aligment, efficient preference optimization, positional indices synthesis

TL;DR: This paper first introduce an efficient preference optimization training strategy for long-context alignment.

Abstract:

Long-context models (LCMs) have shown great potential in processing long input sequences (even more than 100M tokens) conveniently and effectively.
With significant progress, recent research has pointed out that LCMs can accurately locate token-level salient information within the context.
Yet, the generation performance of these LCMs is far from satisfactory and might result in misaligned responses, such as hallucinations.
To enhance the generation capability of LCMs, existing works have investigated the effects of data size and quality for both pre-training and instruction tuning.
Though achieving meaningful improvement, previous methods fall short in either effectiveness or efficiency.
In this paper, we introduce LOGO (Long cOntext aliGnment via efficient preference Optimization), a training strategy that first introduces preference optimization for long-context alignment.
To overcome the GPU memory-bound issue caused by the long sequence, LOGO employs a reference-free preference optimization strategy and adopts a position synthesis method to construct the training data.
By training with only 0.3B data on a single 8$\times$A800 GPU machine for 16 hours, LOGO allows the Llama-3-8B-Instruct-80K model to achieve comparable performance with GPT-4 in real-world long-context tasks while preserving the model's original capabilities on other tasks, e.g., language modeling and MMLU.
Moreover, LOGO can extend the model's context window size while enhancing its generation performance.

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 11006