Basis for Intentions: Efficient Inverse Reinforcement Learning using Past Experience

Marwa Abdulhai, Natasha Jaques, Sergey Levine

Keywords: inverse reinforcement learning, successor features, multi-task reinforcement learning

TL;DR: leveraging prior RL experience for faster inverse RL

Abstract: This paper addresses the problem of inverse reinforcement learning (IRL) – inferring the reward function of an agent from observing its behavior. IRL can provide a generalizable and compact representation for apprenticeship learning, or enable accurately inferring the preferences of a person in order to assist them. However, effective IRL is challenging, because many reward functions can be compatible with an observed behavior. We focus on how prior reinforcement learning (RL) experience can be leveraged to make IRL faster and more efficient. We propose the algorithm BASIS (Behavior Acquisition through Successor-feature Intention inference from Samples), which leverages multi-task RL pre-training and successor features to allow an agent to build a strong basis for intentions that spans the space of possible goals in a given domain. When exposed to just a few expert demonstrations optimizing a novel goal, the agent uses its basis to quickly and effectively infer the reward function. Our experiments reveal that our method is highly effective at inferring and optimizing demonstrated reward functions, accurately inferring reward functions from less than 100 trajectories.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Reinforcement Learning (eg, decision and control, planning, hierarchical RL, robotics)

Community Implementations: [ 3 code implementations](https://www.catalyzex.com/paper/basis-for-intentions-efficient-inverse/code)