PHIRL: Progress Heuristic for Inverse Reinforcement Learning

HANG YU, James Staley, Shijie Fang, Wenchang Gao, Reuben M Aronson, Elaine Short

Keywords: Robot Learning, Learning from Demonstrations, Learning from Human Feedback

TL;DR: In this work, we proposed an Inverse Reinforcement Learning method that learns reward functions by alternately learning a reward function from demonstrations and shaping the learned reward function with a novel human feedback progress.

Abstract: In this paper, we propose a new framework, PHIRL, leveraging both human demonstrations and a recently introduced type of human feedback, \textit{progress}. \textit{Progress} describes the completion rate of a task based on the end state of a trajectory and has been shown to correlate with task success while being consistent across multiple non-experts. We use \textit{progress} to annotate a part of the demonstration dataset. In PHIRL, reward functions are learned using IRL methods and then shaped to align with the \textit{progress} annotations over the annotated demonstrations. Our method does not intensively rely on humans to stay in the learning loop to provide feedback during the training and is capable of mitigating reward hacking and bottleneck issues. We validate PHIRL using a simulation study and a block lifting task. Our results show that PHIRL learns better reward functions and is more robust when the demonstrations are imperfect.

Submission Number: 12