Combining Imitation and Reinforcement Learning with Free Energy Principle

Ryoya Ogishima, Izumi Karino, Yasuo Kuniyoshi

Keywords: Imitation, Reinforcement Learning, Free Energy Principle

Abstract: Imitation Learning (IL) and Reinforcement Learning (RL) from high dimensional sensory inputs are often introduced as separate problems, but a more realistic problem setting is how to merge the techniques so that the agent can reduce exploration costs by partially imitating experts at the same time it maximizes its return. Even when the experts are suboptimal (e.g. Experts learned halfway with other RL methods or human-crafted experts), it is expected that the agent outperforms the suboptimal experts’ performance. In this paper, we propose to address the issue by using and theoretically extending Free Energy Principle, a unified brain theory that explains perception, action and model learning in a Bayesian probabilistic way. We find that both IL and RL can be achieved based on the same free energy objective function. Our results show that our approach is promising in visual control tasks especially with sparse-reward environments.

One-sentence Summary: Extending Free Energy Principle to achieve imitation reinforcement learning for sparse reward problems with suboptimal experts

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Reviewed Version (pdf): /references/pdf?id=KvdHIDeYmK