Live in the Moment: Learning Dynamics Model Adapted to Evolving PolicyDownload PDF

28 May 2022, 15:35 (modified: 23 Jun 2022, 17:19)DARL 2022 SpotlightReaders: Everyone
Keywords: Reinforcement Learning, Model-based Reinforcement Learning, State-action Visitation Distribution, Distribution Shift, Policy-adapted Dynamics Model Learning
TL;DR: We theoretically analyze how the distribution of historical policies affects the model learning and model rollouts and propose a novel model-based RL method, PMAC, which learns a policy-adapted dynamics model based on a policy-adaptation mechanism.
Abstract: Model-based reinforcement learning (RL) achieves higher sample efficiency in practice than model-free RL by learning a dynamics model to generate samples for policy learning. Previous works learn a ``global'' dynamics model to fit the state-action visitation distribution for all historical policies. However, in this paper, we find that learning a global dynamics model does not necessarily benefit model prediction for the current policy since the policy in use is constantly evolving. The evolving policy during training will cause state-action visitation distribution shifts. We theoretically analyze how the distribution of historical policies affects the model learning and model rollouts. We then propose a novel model-based RL method, named \textit{Policy-adaptation Model-based Actor-Critic (PMAC)}, which learns a policy-adapted dynamics model based on a policy-adaptation mechanism. This mechanism dynamically adjusts the historical policy mixture distribution to ensure the learned model can continually adapt to the state-action visitation distribution of the evolving policy. Experiments on a range of continuous control environments in MuJoCo show that PMAC achieves state-of-the-art asymptotic performance and almost two times higher sample efficiency than prior model-based methods.
0 Replies

Loading