Learning from Preferences and Mixed Demonstrations in General Settings

Jason Ross Brown, Carl Henrik Ek, Robert D. Mullins

Keywords: reinforcement learning, rl, human feedback, rlhf, modelling, preferences, demonstrations, rankings, machine learning, reward learning

TL;DR: We introduce a principled method for learning reward functions in RL from preferences, and ranked positive and negative demonstrations.

Abstract:

Reinforcement learning is a general method for learning in sequential settings, but it can often be difficult to specify a good reward function when the task is complex.
In these cases, preference feedback or expert demonstrations can be used instead.
However, existing approaches utilising both together are either ad-hoc or rely on domain-specific properties.
Building upon previous work, we develop a mathematical framework for learning from human data and based on this we introduce LEOPARD: Learning Estimated Objectives from Preferences And Ranked Demonstrations.
LEOPARD can simultaneously learn from a broad range of data, including negative/failed demonstrations, to effectively learn reward functions in general domains.
It does this by modelling the human feedback as reward-rational partial orderings over available trajectories.
We find that when a limited amount of preference and demonstration feedback is available, LEOPARD outperforms baselines by a significant margin.
Furthermore, we use LEOPARD to investigate learning from many types of feedback compared to just a single one, and find that a combination of feedback types is often beneficial.

Primary Area: Reinforcement learning (e.g., decision and control, planning, hierarchical RL, robotics)

Submission Number: 24917