Sim and Real: Better Together

Shirli Di-Castro Shashua, Dotan Di Castro, Shie Mannor

Keywords: Reinforcement Learning, Robotics, Replay buffer

TL;DR: We provide an algorithm and a novel replay buffer analysis for sim to real in robotics application

Abstract: Simulation is used extensively in autonomous systems, particularly in robotic manipulation. By far, the most common approach is to train a controller in simulation, and then use it as an initial starting point for the real system. We demonstrate how to learn simultaneously from both simulation and interaction with the real environment. We propose an algorithm for balancing the large number of samples from the high throughput but less accurate simulation and the low-throughput, high-fidelity and costly samples from the real environment. We achieve that by maintaining a replay buffer for each environment the agent interacts with. We analyze such multi-environment interaction theoretically, and provide convergence properties, through a novel theoretical replay buffer analysis. We demonstrate the efficacy of our method on a sim-to-real environment.

Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.

Code: https://github.com/sdicastro/SimAndRealBetterTogether

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/sim-and-real-better-together/code)