PrivilegedDreamer: Explicit Imagination of Privileged Information for Adaptation in Uncertain Environments
Primary Area: reinforcement learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Keywords: reinforcement learning, adaptation
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Abstract: In many real-world control problems, such as robotics, the system dynamics can be significantly affected by unobservable hidden parameters, like friction coefficients. To represent these kinds of domains, we use Hidden-parameter Markov Decision Processes (HIP-MDPs), which model sequential decision problems where hidden variables affect the transition and reward functions. Existing approaches, such as domain randomization, domain adaptation, and meta-learning, simply treat the effect of hidden parameters as additional variance in dynamics and often struggle to effectively handle HIP-MDP problems, especially when rewards are parameterized by hidden variables. To address this, we introduce PrivilegedDreamer, a model-based reinforcement learning framework that extends Dreamer, a powerful world-modeling approach, by incorporating an explicit parameter estimation module. We introduce a novel dual recurrent architecture that explicitly estimates hidden parameters from limited historical data and enables us to condition the model, actor, and critic networks on these estimated parameters. Our empirical analysis on five diverse HIP-MDP tasks demonstrates that it outperforms state-of-the-art model-based, model-free, and domain adaptation learning algorithms. Furthermore, we also conduct ablation studies to justify our design decisions.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 8093
Loading