ED2: An Environment Dynamics Decomposition Framework for World Model Construction

Cong Wang, Tianpei Yang, Jianye HAO, YAN ZHENG, Hongyao Tang, Fazl Barez, Jinyi Liu, Jiajie Peng, haiyin piao, Zhixiao Sun

Keywords: model based reinforcement learning

Abstract: Model-based reinforcement learning methods achieve significant sample efficiency in many tasks, but their performance is often limited by the existence of the model error. To reduce the model error, previous works use a single well-designed network to fit the entire environment dynamics, which treats the environment dynamics as a black box. However, these methods lack to consider the environmental decomposed property that the dynamics may contain multiple sub-dynamics, which can be modeled separately, allowing us to construct the world model more accurately. In this paper, we propose the Environment Dynamics Decomposition (ED2), a novel world model construction framework that models the environment in a decomposing manner. ED2 contains two key components: sub-dynamics discovery (SD2) and dynamics decomposition prediction (D2P). SD2 discovers the sub-dynamics in an environment and then D2P constructs the decomposed world model following the sub-dynamics. ED2 can be easily combined with existing MBRL algorithms and empirical results show that ED2 significantly reduces the model error and boosts the performance of the state-of-the-art MBRL algorithms on various continuous control tasks.

One-sentence Summary: We proposed a new world model construction framework in decomposing manner