Annotation-Efficient Language Model Alignment via Diverse and Representative Response Texts
Keywords: Language model alignment, Direct preference optimization
TL;DR: We propose Annotation-Efficient Preference Optimization, a method to fucus the annotation budget on diverse and representative responses and show that it outperforms conventional Direct Preference Optimization.
Abstract: Preference optimization is a standard approach to fine-tuning large language models to align with human preferences.
The quantity, diversity, and representativeness of the preference dataset are critical to the effectiveness of preference optimization.
However, obtaining a large amount of preference annotations is difficult in many applications.
This raises the question of how to use the limited annotation budget to create an effective preference dataset.
To this end, we propose Annotation-Efficient Preference Optimization (AEPO).
Instead of exhaustively annotating preference over all available response texts, AEPO selects a subset of responses that maximizes diversity and representativeness from the available responses and then annotates preference over the selected ones.
In this way, AEPO focuses the annotation budget on labeling preference over a smaller subset of responses.
We evaluate the performance of Direct Preference Optimization (DPO) using AEPO and show that it outperforms models trained using a standard DPO with the same annotation budget.
Primary Area: applications to computer vision, audio, language, and other modalities
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 12975
Loading