Effective State Space Exploration with Phase State Graph Generation and Goal-based Path Planning

Sinuo Zhang, Jifeng Hu, Xinqi Du, Zhejian Yang, Yang Yu, Hechang Chen

Abstract: Exploring the state space efficiently is a crucial problem in reinforcement learning as it holds significant importance for learning optimal policies. One effective approach involves learning different sub-policies to cover various sub-spaces of the state space, with each sub-policy corresponding to a specific goal. However, the unevenness of the state probability distribution may lead to exploration difficulties in deep reinforcement learning. To overcome this challenge, we propose a Phase State Graph Exploration framework (PSGE), guiding the agent towards more promising directions for exploration. Specifically, we design a graph-based state space exploration framework to separate the combination space into sub-spaces and define the combination space and evaluation criteria for the agent’s sub-policies. In addition, hypernetwork is leveraged to decouple sub-policies and sub-goals, ensuring diversity among the agent’s sub-policies and reward shaping is used to provide dense internal reward signals for policy training, which encourages the agent to learn more efficiently. Experiments on combining control and navigation tasks demonstrate that PSGE performs well in controlling agent across various difficulty level tasks.