Automated Rewards via LLM-Generated Progress Functions

Vishnu Sarukkai, Brennan Shacklett, Zander Majercik, Kush Bhatia, Christopher Re, Kayvon Fatahalian

Keywords: reinforcement learning, LLMs, foundation models

Abstract:

Large Language Models (LLMs) have the potential to automate reward engineering for Reinforcement Learning (RL) by leveraging their broad domain knowledge across various tasks. However, they often need many iterations of trial-and-error to generate effective reward functions.
This process is costly because evaluating every sampled reward function requires completing
the full policy optimization process for each function.
In this paper, we introduce an LLM-driven reward generation framework that is able to produce state-of-the-art policies on the challenging Bi-DexHands benchmark with 20$\times$ fewer reward function samples than the prior state-of-the-art work.
Our key insight is that we reduce the problem of generating task-specific rewards to the problem of coarsely estimating task progress.
Our two-step solution leverages the task domain knowledge and the code synthesis abilities of LLMs to author progress functions that estimate task progress from a given state.
Then, we use this notion of progress to discretize states, and generate count-based intrinsic rewards using the low-dimensional state space.
We show that the combination of LLM-generated progress functions and count-based intrinsic rewards is essential for our performance gains, while alternatives such as generic hash-based counts or using progress directly as a reward function fall short.

Primary Area: reinforcement learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 5188