A Divergence Minimization Perspective on Imitation Learning Methods

Seyed Kamyar Seyed Ghasemipour, Richard Zemel, Shixiang Shane Gu

Abstract:

In many settings, it is desirable to learn decision-making and control policies through learning or bootstrapping from expert demonstrations. The
most common approaches under this Imitation Learning (IL) framework are Behavioural Cloning (BC), and Inverse Reinforcement Learning (IRL). Recent methods for IRL have demonstrated the capacity to learn effective policies with access to a very limited set of demonstrations, a scenario in which BC methods
often fail. Unfortunately, due to multiple factors of variation, directly comparing
these methods does not provide adequate intuition for understanding this difference in performance. In this work, we present a unified probabilistic perspective
on IL algorithms based on divergence minimization. We present f-MAX, an fdivergence generalization of AIRL [1], a state-of-the-art IRL method. f-MAX
enables us to relate prior IRL methods such as GAIL [2] and AIRL [1], and understand their algorithmic properties. Through the lens of divergence minimization we tease apart the differences between BC and successful IRL approaches,
and empirically evaluate these nuances on simulated high-dimensional continuous control domains. Our findings conclusively identify that IRL’s state-marginal
matching objective contributes most to its superior performance. Lastly, we apply our new understanding of IL method to the problem of state-marginal matching, where we demonstrate that in simulated arm pushing environments we can
teach agents a diverse range of behaviours using simply hand-specified state distributions and no reward functions or expert demonstrations. For datasets and reproducing results please refer to https://github.com/KamyarGh/rl_swiss/
blob/master/reproducing/fmax_paper.md.