Annotation-Efficient Language Model Alignment via Diverse and Representative Response Texts

ACL ARR 2024 December Submission1517 Authors

Abstract:

Preference optimization is a standard approach to fine-tuning large language models to align with human preferences.
The quantity, diversity, and representativeness of the preference dataset are critical to the effectiveness of preference optimization.
However, obtaining a large amount of preference annotations is difficult in many applications.
This raises the question of how to use the limited annotation budget to create an effective preference dataset.
To this end, we propose Annotation-Efficient Preference Optimization (AEPO).
Instead of exhaustively annotating preference over all available response texts, AEPO selects a subset of responses that maximizes diversity and representativeness from the available responses and then annotates preference over the selected ones.
In this way, AEPO focuses the annotation budget on labeling preferences over a smaller but informative subset of responses.
We evaluate the performance of Direct Preference Optimization (DPO) using AEPO and show that it outperforms models trained using a standard DPO with the same annotation budget.

Paper Type: Long

Research Area: Machine Learning for NLP

Research Area Keywords: optimization methods

Contribution Types: NLP engineering experiment, Approaches to low-resource settings

Languages Studied: English, Japanese

Submission Number: 1517