Joint Representations for Reinforcement Learning with Multiple Sensors

Philipp Becker, Sebastian Markgraf, Fabian Otto, Gerhard Neumann

Primary Area: reinforcement learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Representations for RL, Image-Based RL, Contrastive Learning, Reinfrocement Learning

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: Learning joint representations for multiple sensors improves reinforcement learning

Abstract: Combining inputs from multiple sensor modalities effectively in reinforcement learning (RL) is an open problem. While many self-supervised representation learning approaches exist to improve performance and sample complexity for image-based RL, they usually neglect other available information, such as robot proprioception. In this work, we show how using this proprioception for representation learning can help algorithms to focus on relevant aspects and guide them toward finding better representations. Building on Recurrent State Space Models, we systematically analyze representation learning approaches for RL from multiple sensors. We propose a novel combination of reconstruction-based and contrastive losses, which allows us to choose the most appropriate method for each sensor modality, and demonstrate its benefits in a wide range of settings. This evaluation includes model-free and model-based RL on complex tasks where the images contain distractions or occlusions, a new locomotion suite, and a visually realistic mobile manipulation task. We show that learning a joint representation by combining contrastive and reconstruction-based losses significantly improves performance compared to the common practice of combining image representations and proprioception and allows solving more complex tasks that are beyond the reach of current SOTA representation learning methods.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 5814