RLHF and IIA: Perverse Incentives

Wanqiao Xu, Shi Dong, Xiuyuan Lu, Grace Lam, Zheng Wen, Benjamin Van Roy

Keywords: RLHF, independence of irrelevant alternatives, reinforcement learning, language modeling

TL;DR: Existing algorithms for RLHF can incentivize responses at odds with preferences because they are based on models that assume independence of irrelevant alternatives.

Abstract: Existing algorithms for reinforcement learning from human feedback (RLHF) can incentivize responses at odds with preferences because they are based on models that assume independence of irrelevant alternatives (IIA). The perverse incentives induced by IIA hinder innovations on query formats and learning algorithms.

Submission Number: 5