Continual Invariant Image Mapping from Randomized Simulations for Sim2real Transfer in Robotic Manipulation

Xiongyu Xie, Josip Josifovski, Mohammadhossein Malmir, Alois Knoll

Keywords: Continual Learning, Robotics, Simulation, Sim2Real, Domain Randomization, Variational Autoencoder, Reinforcement Learning

TL;DR: We combine randomization with continual learning of invariant image representation for a specific robotic task to perform sim2real transfer and mitigate problems like changes in the underlying distribution or subsequent model retraining from scratch.

Abstract: Currently, deep reinforcement learning algorithms require large amounts of training data to learn a specific task, which makes them infeasible to train directly on real robotic systems. To overcome this obstacle, one usually relies on training in simulation and randomizes aspects of the simulation to compensate for the mismatch between the simulator and the real system. However, it is not always clear which aspect of the simulation requires randomization and usually enabling an additional randomization parameter or simulation modifications require model retraining from scratch. To address this problem, in this paper we explore how continual state representation learning can be combined with parameter randomization for vision-based reinforcement learning of robotic tasks, to minimize the need for complete model retraining. To this end, we use variational autoencoder (VAE) to continually learn to reconstruct invariant image representation from sequentially randomized/augmented simulation images. Independently, a reinforcement learning model is trained on the invariant image representation to solve a robotic manipulation task. Then, the VAE is used to translate randomized/augmented simulation images or real-world images to the invariant representation images on which the RL agent can operate. Initial results show that the VAE can continually learn reconstruction to invariant images and it can also be used to bridge the sim2real gap by reconstructing correctly real camera images.