Imitation with Neural Density Models

Kuno Kim, Akshat Jindal, Yang Song, Jiaming Song, Yanan Sui, Stefano Ermon

Keywords: Imitation Learning, Reinforcement Learning, Density Estimation, Density Model, Maximum Entropy RL, Mujoco

Abstract: We propose a new framework for Imitation Learning (IL) via density estimation of the expert's occupancy measure followed by Maximum Occupancy Entropy Reinforcement Learning (RL) using the density as a reward. Our approach maximizes a non-adversarial model-free RL objective that provably lower bounds reverse Kullback–Leibler divergence between occupancy measures of the expert and imitator. We present a practical IL algorithm, Neural Density Imitation (NDI), which obtains state-of-the-art demonstration efficiency on benchmark control tasks.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: New Imitation Learning framework based on density estimation that achieves good demonstration efficiency

Community Implementations: [ 5 code implementations](https://www.catalyzex.com/paper/arxiv:2010.09808/code)

Reviewed Version (pdf): https://openreview.net/references/pdf?id=ecH5raXH3V