Utility-inspired Reward Transformations Improve Reinforcement Learning Training of Language Models

Roberto-Rafael Maura-Rivero, Chirag Nagpal, Roma Patel, Francesco Visin

Keywords: Reinforcement Learning, Language Models, Reward Functions, Utility Theory, RLHF

TL;DR: This paper proposes a transformation inspired by economic utility theory, which reshapes individual reward signals amplifying penalties for critically low rewards before aggregating them in RLHF.

Abstract:

Current methods that train large language models (LLMs) with reinforcement learning feedback, often resort to averaging outputs of multiple rewards functions during training. This overlooks crucial aspects of individual reward dimensions and inter-reward dependencies that can lead to sub-optimal outcomes in generations.
In this work, we show how linear aggregation of rewards exhibits some vulnerabilities that can lead to undesired properties of generated text.
We then propose a transformation of reward functions inspired by economic theory of utility functions (specifically Inada conditions), that enhances sensitivity to low reward values while diminishing sensitivity to already high values. We compare our approach to the existing baseline methods that linearly aggregate rewards and show how the Inada-inspired reward feedback is superior to traditional weighted averaging. We quantitatively and qualitatively analyse the difference in methods, and see that models trained with Inada-transformations score as more helpful and less harmful than baselines.

Journal Edition Interest: No

Submission Number: 11