Coherent Soft Imitation Learning

Joe Watson, Sandy Huang, Nicolas Heess

Keywords: imitation learning, inverse reinforcement learning, behavioral cloning, learning from demonstration

TL;DR: Perform imitation learning by using a behavioural cloning policy to construct an estimate of a shaped reward function.

Abstract:

Imitation learning methods seek to learn from an expert either through behavioral cloning (BC) for the policy or inverse reinforcement learning (IRL) for the reward.
Such methods enable agents to learn complex tasks from humans that are difficult to capture with hand-designed reward functions.
Choosing between BC or IRL for imitation depends on the quality and state-action coverage of the demonstrations, as well as additional access to the Markov decision process.
Hybrid strategies that combine BC and IRL are rare, as initial policy optimization against inaccurate rewards diminishes the benefit of pretraining the policy with BC.
Our work derives an imitation method that captures the strengths of both BC and IRL.
In the entropy-regularized (`soft') reinforcement learning setting, we show that the behavioral-cloned policy can be used as both a shaped reward and a critic hypothesis space by inverting the regularized policy update.
This coherency facilitates fine-tuning cloned policies using the reward estimate and additional interactions with the environment.
This approach conveniently achieves imitation learning through initial behavioral cloning and subsequent refinement via RL with online or offline data sources.
The simplicity of the approach enables graceful scaling to high-dimensional and vision-based tasks, with stable learning and minimal hyperparameter tuning, in contrast to adversarial approaches.
For the open-source implementation and simulation results, see https://joemwatson.github.io/csil/.

Submission Number: 7098