Task-Relevant Adversarial Imitation Learning

Konrad Zolna, Scott Reed, Alexander Novikov, Ziyu Wang, Sergio Gómez, David Budden, Serkan Cabi, Misha Denil, Nando de Freitas

TL;DR: Improve GAIL by preventing the discriminator from exploiting task-irrelevant information, to solve difficult sim robot manipulation tasks from pixels.

Abstract: We show that a critical problem in adversarial imitation from high-dimensional sensory data is the tendency of discriminator networks to distinguish agent and expert behaviour using task-irrelevant features beyond the control of the agent. We analyze this problem in detail and propose a solution as well as several baselines that outperform standard Generative Adversarial Imitation Learning (GAIL). Our proposed solution, Task-Relevant Adversarial Imitation Learning (TRAIL), uses a constrained optimization objective to overcome task-irrelevant features. Comprehensive experiments show that TRAIL can solve challenging manipulation tasks from pixels by imitating human operators, where other agents such as behaviour cloning (BC), standard GAIL, improved GAIL variants including our newly proposed baselines, and Deterministic Policy Gradients from Demonstrations (DPGfD) fail to find solutions, even when the other agents have access to task reward.

Keywords: adversarial, imitation, robot, manipulation