Goal Achievement Guided Exploration: Mitigating Premature Convergence in Learning Robot Control

Shengchao Yan, Baohe Zhang, Joschka Boedecker, Wolfram Burgard

Keywords: Robots, Reinforcement Learning, Exploration

Abstract: Premature convergence to suboptimal policies remains a significant challenge in reinforcement learning (RL), particularly for robots with many degrees of freedom and in tasks with non-convex reward landscapes. Existing work usually utilizes reward shaping to encourage exploring promising spaces. However, this may inadvertently introduce new local optima and impair the optimization for the actual target reward. To address this issue, we propose Goal Achievement Guided Exploration (GAGE), a novel approach that incorporates an agent's goal achievement as a dynamic criterion for balancing exploration and exploitation. GAGE adaptively adjusts the exploitation level based on the agent's current performance relative to an estimated optimal performance, thereby mitigating premature convergence. Extensive evaluations demonstrate that GAGE substantially improves learning outcomes across various challenging whole-body control tasks by adapting convergence based on task success. GAGE can seamlessly integrate into existing RL frameworks, highlighting its potential as a versatile tool for enhancing exploration strategies in RL for robot control.

Submission Number: 10