Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment

Jiazheng Zhang, Wenqing Jing, Zizhuo Zhang, Zhiheng Xi, Shihan Dou, Rongxiang Weng, Jiahuan Li, Jingang Wang, Mingxu Chai, Shibo Hong, Yuming Yang, Xuanjing Huang, Tao Gui, Qi Zhang

Keywords: Preference Learning, Reward Modeling, RLHF

Abstract:

Reward models (RMs) play a pivotal role in aligning large language models (LLMs) with human values.
However, noisy preferences in human feedback can lead to reward misgeneralization – a phenomenon where reward models learn spurious correlations or overfit to noisy preferences, which poses important challenges to the generalization of RMs.
This paper systematically analyzes the characteristics of preference pairs and aims to identify how noisy preferences differ from human-aligned preferences in reward modeling.
Our analysis reveals that noisy preferences are difficult for RMs to fit, as they cause sharp training fluctuations and irregular gradient updates.
These distinctive dynamics suggest the feasibility of identifying and excluding such noisy preferences.
Empirical studies clarify that policy LLM optimized with a reward model trained on the full preference dataset, which includes substantial noise, performs worse than the one trained on a subset of exclusively high-quality data.
To address this challenge, we propose an online Collaborative Reward Modeling (CRM) framework to achieve robust preference learning through peer review and curriculum learning.
In particular, CRM maintains two RMs that collaboratively filter potential noisy preferences by peer-reviewing each other’s data selections.
Curriculum learning establishes a well-defined learning trajectory to synchronize the capabilities of two RMs, further promoting the utility of peer review.
Extensive experiments demonstrate that CRM significantly enhances RM generalization, with up to $9.94$-points improvement on RewardBench under an extreme 40\% noise.
Moreover, CRM can seamlessly extend to implicit-reward alignment methods, offering a robust and versatile alignment strategy.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 5067