Comparison-based Active Preference Learning for Multi-dimensional Personalization

ACL ARR 2025 February Submission1913 Authors

Abstract: Large language models (LLMs) have shown remarkable success, but aligning them with human preferences remains a core challenge. As individuals have their own, multi-dimensional preferences, recent studies have explored multi-dimensional personalization, which aims to enable models to generate responses personalized to explicit preferences. However, human preferences are often implicit and thus difficult to articulate, limiting the direct application of this approach. To bridge this gap, we introduce a comparison-based active preference learning framework to capture implicit user preferences. Building on Bayesian inference, our work introduces a modified posterior update procedure to mitigate estimation bias and potential noise in comparisons. Also, inspired by generalized binary search, we employ an active query selection strategy to minimize the number of required comparisons by a user. Through theoretical analysis and experiments on language generation tasks, we demonstrate feedback efficiency and effectiveness of our framework in personalizing model responses.

Paper Type: Long

Research Area: Machine Learning for NLP

Research Area Keywords: human-in-the-loop / active learning, personalization, multi-objective alignment

Languages Studied: English

Submission Number: 1913