Learning from Preferences and Mixed Demonstrations in General Settings
Keywords: reinforcement learning, rl, human feedback, rlhf, modelling, preferences, demonstrations, rankings, machine learning, reward learning
TL;DR: We introduce a principled method for learning reward functions in RL from preferences, and ranked positive and negative demonstrations.
Abstract: Reinforcement learning is a general method for learning in sequential settings, but it can often be difficult to specify a good reward function when the task is complex.
In these cases, preference feedback or expert demonstrations can be used instead.
However, existing approaches utilising both together are either ad-hoc or rely on domain-specific properties.
Building upon previous work, we develop a novel theoretical framework for learning from human data.
Based on this we introduce LEOPARD: Learning Estimated Objectives from Preferences And Ranked Demonstrations.
LEOPARD can simultaneously learn from a broad range of data, including negative/failed demonstrations, to effectively learn reward functions in general domains.
It does this by modelling the human feedback as reward-rational partial orderings over available trajectories.
We find that when a limited amount of human feedback is available, LEOPARD outperforms the current standard practice of pre-training on demonstrations and finetuning on preferences, as well as other baselines.
Furthermore, we show that LEOPARD learns faster when given many types of feedback, rather than just a single one.
Primary Area: reinforcement learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 12169
Loading