Reward Model Boosting for RLHF

Jiabin Fan, Dezhi Ye, Yongchang Hao, Lili Mou

Keywords: Reinforcement Learning, Machine Learning, RLHF, Reward Overoptimization, Reward Hacking

TL;DR: We propose Reward Model Boosting (RMB), which uses advanced boosting to construct a reliable reward signal for RLHF, effectively mitigating reward hacking and significantly improving policy learning.

Abstract: Reinforcement Learning from Human Feedback (RLHF) is a powerful technique for aligning large language models (LLMs) with human preference. However, it often suffers from the reward hacking issue, where policy optimization improves the proxy reward model while actually degrading performance with respect to the true human preference, due to the imperfection of the proxy. To address this, we propose Reward Model Boosting (RMB), a novel approach that enhances the robustness and reliability of the reward signal for RLHF. RMB first trains a set of reward models with a diverse-promoting regularizer. This encourages each model to learn complementary aspects of the reward landscape. Then, RMB learns a lightweight aggregator in the principle of boosting to aggregate the outputs of the diverse reward models into a more accurate and robust reward signal. Our extensive experiments demonstrate that RMB significantly improves reward accuracy on both in-distribution and out-of-distribution datasets, substantially mitigating the reward hacking issue and ultimately improving RLHF performance.

Primary Area: other topics in machine learning (i.e., none of the above)

Submission Number: 4599