ManiSkill3: GPU Parallelized Robot Simulation and Rendering for Generalizable Embodied AI

Stone Tao, Fanbo Xiang, Arth Shukla, Yuzhe Qin, Xander Hinrichsen, Xiaodi Yuan, Chen Bao, Xinsong Lin, Yulin Liu, Tse-Kai Chan, Yuan Gao, Xuanlin Li, Tongzhou Mu, Nan Xiao, Arnav Gurha, Viswesh N, Yong Woo Choi, Yen-Ru Chen, Zhiao Huang, Roberto Calandra et al. (3 additional authors not shown)

Track: system paper (must be submitted with a supplementary video)

Keywords: simulation, sim2real, real2sim, reinforcement learning, imitation learning, gpu simulation

TL;DR: A GPU parallelized robotics simulation and rendering framework/benchmark that provides easy to use APIs, fast visual RL, sim2real and real2sim environments, and a diverse range of prebuilt tasks.

Abstract: Simulation has enabled unprecedented compute-scalable approaches to robotics. However, many existing simulators typically support a narrow range of tasks and lack features critical for scaling generalizable robotics and sim2real. We introduce ManiSkill3, a state-of-the-art state-visual GPU parallelized robotics simulator with contact-rich physics targeting generalizable manipulation. ManiSkill3 supports GPU parallelization of many aspects including simulation+rendering, heterogeneous simulation, pointclouds, and more. GPU simulation+rendering uses 2-4x less GPU memory compared to other platforms and achieves up to 30,000+ FPS in benchmarked environments due to minimal overhead, simulation on the GPU, and the use of the SAPIEN parallel rendering system, enabling visual RL to solve tasks in minutes instead of hours. We further provide the most comprehensive range of tasks spanning 12 distinct domains including but not limited to mobile manipulation, drawing, humanoids, and dextrous manipulation in realistic scenes designed by artists or real-world digital twins. In addition, millions of demonstration frames are provided from motion planning, RL, and teleoperation. ManiSkill3 also provides a comprehensive set of baselines that span popular RL and learning-from-demonstrations algorithms.

Anonymization: This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs.

Presenter: ~Stone_Tao1

Format: Yes, the presenting author will attend in person if this work is accepted to the workshop.

Funding: No, the presenting author of this submission does *not* fall under ICLR’s funding aims, or has sufficient alternate funding.

Submission Number: 12