Identifiable State Disentanglement for Reinforcement Learning with Policy Optimality
Keywords: Reinforcement Learning, World Model, Disentanglement
Abstract: Recent progress in reinforcement learning (RL) has demonstrated significant efficacy in the optimization of policies, in the presence of noise. These works, however, neglect the importance of independence between signal and noise in latent space, resulting in limited performance. To address this concern, we first analyze the identifiability result of latent signal and latent noise, which implicitly highlights that the independence between latent signal and latent noise is crucial in RL. We then convert the identifiability result into a novel method, which isolates signal from noise in latent space, by effectively integrating structural independence and statistical independence into a unified framework. Structurally, the proposed method respectively employs two different decoders on latent signal and latent noise, so that each decoder captures exclusive features specific to its own space. Statistically, the independence between latent signal and latent noise is enforced by a correlation-coefficients objective. Experiments on extensive benchmark control tasks demonstrate that the proposed approach surpasses existing algorithms in effectively disentangling signals from noise.
Primary Area: reinforcement learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 5411
Loading