Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignment

Weichao Zhou, Wenchao Li

Keywords: inverse reinforcement learning, imitation learning, reinforcement learning

TL;DR: Propose a new formulation for inverse reinforcement learning-based imitation learning to mitigate the task-reward misalignment.

Abstract:

Many imitation learning (IL) algorithms use inverse reinforcement learning (IRL) to infer a reward function that aligns with the demonstration.
However, the inferred reward functions often fail to capture the underlying task objectives.
In this paper, we propose a novel framework for IRL-based IL that prioritizes task alignment over conventional data alignment. Our framework is a semi-supervised approach that leverages expert demonstrations as weak supervision to derive a set of candidate reward functions that align with the task rather than only with the data. It then adopts an adversarial mechanism to train a policy with this set of reward functions to gain a collective validation of the policy's ability to accomplish the task. We provide theoretical insights into this framework's ability to mitigate task-reward misalignment and present a practical implementation. Our experimental results show that our framework outperforms conventional IL baselines in complex and transfer learning scenarios.

Primary Area: Reinforcement learning

Submission Number: 19095