Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning

Sriyash Poddar, Yanming Wan, Hamish Ivison, Abhishek Gupta, Natasha Jaques

Keywords: reinforcement learning from human feedback, preference based reward learning, pluralistic alignment, multimodal reward modeling, variational inference

Abstract: Current Reinforcement Learning from Human Feedback (RLHF) techniques cannot account for differences in human preferences across a diverse population. When these differences arise, these frameworks average over them, leading to inaccurate rewards and poor performance for individual subgroups. To address the need for pluralistic alignment, we develop a class of multimodal RLHF methods based on a latent variable formulation - inferring a novel user-specific latent and learning reward models and policies conditioned on this latent without additional user-specific data. While conceptually simple, we show that in practice, this reward modeling requires careful algorithmic considerations around model architecture and reward scaling. To empirically validate our proposed technique, we first show that it can provide a way to combat under-specification in simulated control problems, inferring and optimizing user-specific reward functions. Next, we conduct experiments on pluralistic language datasets representing diverse user preferences and demonstrate improved reward function accuracy. We additionally show the benefits of this probabilistic framework in actively learning user preferences. This work enables learning from diverse populations, an important challenge naturally occurring in problems from robot learning to foundation model alignment.

Submission Number: 29