ROIL: Robust Offline Imitation Learning without Trajectories

Gersi Doko, Guang Yang, Daniel S. Brown, Marek Petrik

Keywords: Imitation learning, MDPs, robust optimization

TL;DR: Robust Offline Imitation Learning without Trajectories

Abstract: We study the problem of imitation learning via inverse reinforcement learning where the agent attempts to learn an expert's policy from a dataset of collected state, action tuples. We derive a new Robust model-based Offline Imitation Learning method (ROIL) that mitigates covariate shift by avoiding estimating the expert's occupancy frequency. Frequently in offline settings, there is insufficient data to reliably estimate the expert's occupancy frequency and this leads to models that do not generalize well. Our proposed approach, ROIL, is a method that is guaranteed to recover the expert's occupancy frequency and is efficiently solvable as an LP. We demonstrate ROIL's ability to achieve minimal regret in large environments under covariate shift, such as when the state visitation frequency of the demonstrations does not come from the expert.

Submission Number: 83