Versatile Offline Imitation Learning via State-Occupancy Matching

Yecheng Jason Ma, Andrew Shen, Dinesh Jayaraman, Osbert Bastani

Keywords: Offline Imitation Learning, Offline Reinforcement Learning, Learning from Observations, Example-Based Reinforcement Learning, Convex Optimization

TL;DR: A versatile offline imitation learning algorithm that can learn from observations, mismatched experts, and examples.

Abstract: We propose State Matching Offline DIstribution Correction Estimation (SMODICE), a novel and versatile algorithm for offline imitation learning (IL) via state-occupancy matching. Without requiring access to expert actions, SMODICE can be effectively applied to three offline IL settings: (i) imitation from observations (IfO), (ii) IfO with dynamics or morphologically mismatched expert, and (iii) example-based reinforcement learning, which we show can be formulated as a state-occupancy matching problem. We show that the SMODICE objective admits a simple optimization procedure through an application of Fenchel duality, reducing a nested optimization problem to a sequence of stable supervised learning problems. We extensively evaluate SMODICE on both gridworld environments as well as on high-dimensional offline benchmarks. Our results demonstrate that SMODICE is effective for all three problem settings and significantly outperforms prior state-of-art.