Learning Fused State Representations for Control from Multi-View Observations

Zeyu Wang, Yao-Hui Li, Xin Li, Hongyu Zang, Romain Laroche, Riashat Islam

TL;DR: We proposed a method to learn fused state representations for multi-view RL.

Abstract: Multi-View Reinforcement Learning (MVRL) seeks to provide agents with multi-view observations, enabling them to perceive environment with greater effectiveness and precision. Recent advancements in MVRL focus on extracting latent representations from multiview observations and leveraging them in control tasks. However, it is not straightforward to learn compact and task-relevant representations, particularly in the presence of redundancy, distracting information, or missing views. In this paper, we propose **M**ulti-view **F**usion **S**tate for **C**ontrol (**MFSC**), firstly incorporating bisimulation metric learning into MVRL to learn task-relevant representations. Furthermore, we propose a multiview-based mask and latent reconstruction auxiliary task that exploits shared information across views and improves MFSC’s robustness in missing views by introducing a mask token. Extensive experimental results demonstrate that our method outperforms existing approaches in MVRL tasks. Even in more realistic scenarios with interference or missing views, MFSC consistently maintains high performance. The project code is available at [https://github.com/zpwdev/MFSC](https://github.com/zpwdev/MFSC).

Lay Summary:

Robots often rely on cameras and sensors to understand the world around them. Using multiple views can improve their perception, but it also brings new challenges—such as too much redundant data, distractions from irrelevant details, or missing angles altogether.

Our method, called MFSC (Multi-view Fusion State for Control), helps robots make sense of all these different views. It condenses the visual information into a compact summary that focuses only on what’s important for making decisions. This helps robots ignore the noise and concentrate on what truly matters.

We tested MFSC in complex environments, where some views were missing or noisy. Experimental results show that robots using MFSC maintained better decision-making performance in scenarios with missing or noisy views.

Link To Code: https://github.com/zpwdev/MFSC

Primary Area: Reinforcement Learning

Keywords: reinforcement learning, multi-view learning

Submission Number: 1488