IL-SOAR : Imitation Learning with Soft Optimistic Actor cRitic

Stefano Viel, Luca Viano, Volkan Cevher

Abstract:

This paper introduces the SOAR framework for imitation learning. SOAR is an algorithmic template
which learns a policy from expert demonstrations with a primal-dual style algorithm which alternates cost and policy updates. Within the policy updates the SOAR framework prescribe to use an actor critic method with multiple critics to estimate the critic uncertainty and therefore build an optimistic critic fundamental to drive exploration.
When instantiated to the tabular setting, we get a provable algorithms dubbed FRA with guarantees matching the best known results in $\epsilon$.
Practically, the SOAR template is shown to boost consistently the performance of primal dual IL algorithms building on actor critic routines for the policy updates. Approximately, thanks to SOAR, the required number of episodes to achieve the same performance is reduced by a half.

Lay Summary:

Imitation learning is a way for artificial intelligence (AI) systems to learn new skills by watching and copying expert behavior—much like how a child learns by observing adults. However, making AI learn efficiently from demonstrations can be challenging, especially in complex environments.

This paper introduces a new framework called SOAR (Soft Optimistic Actor cRitic) to improve how AI learns from experts. The key idea behind SOAR is to help the AI not just copy what it sees, but also to explore actions it’s less sure about, guided by a sense of “optimism” about what might work well. This is achieved by using multiple “critics” (advisors) within the AI that estimate how good different actions might be, and then encouraging the AI to try actions where these critics are most optimistic.

The authors show that SOAR can be used as a flexible template, improving several popular imitation learning algorithms. In practical tests with simulated robots (using the MuJoCo environment), SOAR helped these algorithms learn faster and more efficiently—cutting the amount of training needed by half to reach the same level of performance.

In summary, SOAR is a promising step towards making AI systems better at learning from demonstrations, allowing them to master new tasks more quickly and with less data.

Primary Area: Reinforcement Learning->Inverse

Keywords: provably efficient IL, SAC based IL, Deep IL

Submission Number: 3930