Towards Personalized Language Models via Inference-time Human Preference Optimization

Published: 10 Oct 2024, Last Modified: 19 Nov 2024AFM 2024 PosterEveryoneRevisionsBibTeXCC BY 4.0
Keywords: LLM Alignment, Personal Preference Optimization, Personalized Alignment
TL;DR: We introduce a new approach for personalized alignment of LLMs based on decode-time frameworks.
Abstract: The impressive generative capabilities of large language models (LLMs) have led to their widespread adoption across diverse applications. However, existing alignment methods, which rely heavily on expensive fine-tuning processes, focus on optimizing for the \textit{general human preferences} such as safety, fairness, and trustworthiness. These approaches suffer from scalability and adaptability issues when addressing \textit{personal preferences} which could be different across users. In this paper, we introduce a novel approach to LLM alignment for personalized preference based on decode-time frameworks. Our approach enables dynamic adaptation to personal preferences during inference, providing a flexible and computationally efficient solution for personalization without the need of training-time interventions. We demonstrate the efficacy of our method on benchmark datasets and tasks, by enhancing LLMs' ability to adapt to diverse personal preferences compared to the existing alignment methods.
Submission Number: 148
Loading