Go to ICLR 2025 Conference homepageDoes RLHF Scale? Exploring the Effects of Data, Model, and Method
Keywords: Language model, Reinforcement learning from human feedback, Scaling
TL;DR: We explore the scaling properties of reinforcement learning from human feedback (RLHF), and show that current RLHF scales less efficiently.
Abstract: This study explores the scaling properties of Reinforcement Learning from Human Feedback (RLHF) in Large Language Models (LLMs).
Although RLHF is considered an important step in the post-training of LLMs, its scaling potential is still largely unknown.
We systematically analyze key components in the RLHF framework—model size, data composition, and inference budget—and their impacts on performance.
Our findings show that increasing data diversity and volume improves reward model performance, helping process-supervision models scale better.
For policy training, more response samples per prompt boost performance initially but quickly plateau.
And larger reward models offer modest gains in policy training.
In addition, larger policy models benefit less from RLHF with a fixed reward model.
Overall, RLHF scales less efficiently than pretraining, with diminishing returns from additional computational resources.
Based on these observations, we propose strategies to optimize RLHF performance within computational limits.
Primary Area: foundation or frontier models, including LLMs
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 9480
Loading