Robust Robotic Control from Pixels using Contrastive Recurrent State-Space Models

Nitish Srivastava, Walter Talbott, Martin Bertran Lopez, Shuangfei Zhai, Joshua M. Susskind

Keywords: contrastive learning, model-based RL, distractions, predictive coding

Abstract: Modeling the world can benefit robot learning by providing a rich training signal for shaping an agent's latent state space. However, learning world models in unconstrained environments over high-dimensional observation spaces such as images is challenging. One source of difficulty is the presence of irrelevant but hard-to-model background distractions, and unimportant visual details of task-relevant entities. We address this issue by learning a recurrent latent dynamics model which contrastively predicts the next observation. This simple model leads to surprisingly robust robotic control even with simultaneous camera, background, and color distractions. We outperform alternatives such as bisimulation methods which impose state-similarity measures derived from divergence in future reward or future optimal actions. We obtain state-of-the-art results on the Distracting Control Suite, a challenging benchmark for pixel-based robotic control.

One-sentence Summary: RSSMs with contrastive prediction work surprisingly well for pixel-based RL even in the presence of severe visual distractions.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2112.01163/code)