Versatile Offline Imitation Learning via State-Occupancy MatchingDownload PDF

04 Mar 2022, 07:18 (edited 16 Apr 2022)ICLR 2022 GPL PosterReaders: Everyone
  • Keywords: Offline Imitation Learning, Offline Reinforcement Learning, Learning from Observations, Example-Based Reinforcement Learning, Convex Optimization
  • TL;DR: A versatile offline imitation learning algorithm that can learn from observations, mismatched experts, and examples.
  • Abstract: We propose State Matching Offline DIstribution Correction Estimation (SMODICE), a novel and versatile algorithm for offline imitation learning (IL) via state-occupancy matching. Without requiring access to expert actions, SMODICE can be effectively applied to three offline IL settings: (i) imitation from observations (IfO), (ii) IfO with dynamics or morphologically mismatched expert, and (iii) example-based reinforcement learning, which we show can be formulated as a state-occupancy matching problem. We show that the SMODICE objective admits a simple optimization procedure through an application of Fenchel duality, reducing a nested optimization problem to a sequence of stable supervised learning problems. We extensively evaluate SMODICE on both gridworld environments as well as on high-dimensional offline benchmarks. Our results demonstrate that SMODICE is effective for all three problem settings and significantly outperforms prior state-of-art.
1 Reply