KL-Regularised Q-Learning: A Token-level Action-Value perspective on Online RLHF

Lennie Wells, Edward James Young, Jason Ross Brown, Sergio Bacallado

Keywords: RLHF, PPO, action-value RL, online RL, human feedback

TL;DR: We propose a novel token-level action-value methodology for online RLHF, which has a revealing theoretical connection to PPO.

Abstract:

Proximal Policy Optimisation (PPO) is an established and effective policy gradient algorithm used for Language Model Reinforcement Learning from Human Feedback (LM-RLHF).
PPO performs well empirically but has a heuristic motivation and handles the KL-divergence constraint used in LM-RLHF in an ad-hoc manner.
In this paper, we develop a a new action-value RL method for the LM-RLHF setting, KL-regularised Q-Learning (KLQ).
We then show that our method is equivalent to a version of PPO in a certain specific sense, despite its very different motivation.
Finally, we benchmark KLQ on two key language generation tasks---summarisation and single-turn dialogue. We demonstrate that KLQ performs on-par with PPO at optimising the LM-RLHF objective, and achieves a consistently higher win-rate against PPO on LLM-as-a-judge evaluations.

Submission Number: 72