Imitation Learning with Sinkhorn Distances

Georgios Papagiannis, Yunpeng Li

Abstract: Imitationlearningalgorithmshavebeeninterpretedasvariantsofdi- vergence minimization problems. The ability to compare occupancy measures between experts and learners is crucial in their effectiveness in learning from demonstrations. In this paper, we present tractable solutions by formulating im- itation learning as minimization of the Sinkhorn distance between occupancy measures. The formulation combines the valuable properties of optimal transport metrics in comparing non-overlapping distributions with a cosine distance cost defined in an adversarially learned feature space. This leads to a highly discrimi- native critic network and optimal transport plan that subsequently guide imitation learning. We evaluate the proposed approach using both the reward metric and the Sinkhorn distance metric on a number of MuJoCo experiments. For the im- plementation and reproducing results please refer to the following repository https://github.com/gpapagiannis/sinkhorn-imitation.