Disagreement-Regularized Imitation LearningDownload PDF

25 Sept 2019, 19:26 (modified: 10 Feb 2022, 11:46)ICLR 2020 Conference Blind SubmissionReaders: Everyone
Original Pdf: pdf
Code: [![github](/images/github_icon.svg) xkianteb/dril](https://github.com/xkianteb/dril) + [![Papers with Code](/images/pwc_icon.svg) 1 community implementation](https://paperswithcode.com/paper/?openreview=rkgbYyHtwB)
Keywords: imitation learning, reinforcement learning, uncertainty
TL;DR: Method for addressing covariate shift in imitation learning using ensemble uncertainty
Abstract: We present a simple and effective algorithm designed to address the covariate shift problem in imitation learning. It operates by training an ensemble of policies on the expert demonstration data, and using the variance of their predictions as a cost which is minimized with RL together with a supervised behavioral cloning cost. Unlike adversarial imitation methods, it uses a fixed reward function which is easy to optimize. We prove a regret bound for the algorithm which is linear in the time horizon multiplied by a coefficient which we show to be low for certain problems in which behavioral cloning fails. We evaluate our algorithm empirically across multiple pixel-based Atari environments and continuous control tasks, and show that it matches or significantly outperforms behavioral cloning and generative adversarial imitation learning.
12 Replies

Loading