Adaptive Multi-model Fusion Learning for Sparse-Reward Reinforcement Learning

Giseung Park, Whiyoung Jung, Sungho Choi, Youngchul Sung

Keywords: sparse-reward RL, intrinsic reward generation, adaptive fusion, information geometry, scale-free property

Abstract: In this paper, we consider intrinsic reward generation for sparse-reward reinforcement learning based on model prediction errors. In typical model-prediction-error-based intrinsic reward generation, an agent has a learning model for the underlying environment. Then intrinsic reward is designed as the error between the model prediction and the actual outcome of the environment, based on the fact that for less-visited or non-visited states, the learned model yields larger prediction errors, promoting exploration helpful for reinforcement learning. This paper generalizes this model-prediction-error-based intrinsic reward generation method to multiple prediction models. We propose a new adaptive fusion method relevant to the multiple-model case, which learns optimal prediction-error fusion across the learning phase to enhance the overall learning performance. Numerical results show that for representative locomotion tasks, the proposed intrinsic reward generation method outperforms most of the previous methods, and the gain is significant in some tasks.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: We propose a new optimal adaptive fusion algorithm for intrinsic reward generation in sparse-reward RL.

Reviewed Version (pdf): https://openreview.net/references/pdf?id=Y38gOER8RW