GaussGym: An open-source real-to-sim framework for learning locomotion from pixels

Alejandro Escontrela, Justin Kerr, Jonas Frey, Yan Duan, Carmelo Sferrazza, Pieter Abbeel

Keywords: simulation, photoreal, robotics, real2sim, sim2real

TL;DR: We integrate 3D Gaussian Splatting into fast vectorized simulators, achieving photorealism at over 1M FPS on consumer GPUs and reducing the sim-to-real gap across diverse tasks.

Abstract: We present a photorealistic robot simulator that integrates 3D Gaussian Splatting as a drop-in renderer within vectorized physics simulators such as IsaacGym. This enables unprecedented speed—exceeding 100,000 steps per second on consumer GPUs—while maintaining high visual fidelity, which we showcase across diverse tasks. We additionally demonstrate its applicability in a sim-to-real robotics setting. Beyond depth-based sensing, our results highlight how rich visual semantics improve navigation and decision-making, such as avoiding undesirable regions. We further showcase the ease of incorporating thousands of environments from iPhone scans, large-scale scene datasets (e.g., GrandTour, ARKit), and outputs from generative video models like Veo, enabling rapid creation of realistic training worlds. This work bridges high-throughput simulation and high-fidelity perception, advancing scalable and generalizable robot learning, and allowing researchers to benchmark their visual locomotion algorithms. All code and data will be open-sourced for the community to build upon. Videos, code, and data are available on the project website: https://gauss-gym.com

Primary Area: datasets and benchmarks

Submission Number: 23922