A Real2Sim2Real Method for Robust Object Grasping with Neural Surface Reconstruction

Luobin Wang, Runlin Guo, Quan Vuong, Yuzhe Qin, Hao Su, Henrik I. Christensen

Abstract: We explore an emerging technique, geometric Real2Sim2Real, in the context of object manipulation. We hypothesize that recent 3D modeling methods provides a path towards building digital replicas of real-world scenes that afford physical simulation and support robust manipulation algorithm learning. Since 6 DOF grasping is one the most important primitives for all manipulation tasks, we study whether geometric Real2Sim2Real can help us train a robust grasping network with high sample efficiency. We propose to reconstruct high-quality meshes from real-world point clouds using state-of-the-art neural surface reconstruction method (the Real2Sim step). Because most simulators take meshes for fast simulation, the reconstructed meshes enable grasp pose labels generation without human efforts. The generated labels can train grasp network that performs robustly in real evaluation scenes (the Sim2Real step). In synthetic and real experiments, we show that the Real2Sim2Real pipeline performs better than baseline grasp networks trained with a ${1}0^{4}\times$ larger dataset by mimicking geometric shapes of target objects in simulation. We also show that our method has better sample efficiency than training the grasping network with a retrieval-based scene reconstruction method. The benefit of the Real2Sim2Real pipeline comes from 1) decoupling scene modeling and grasp sampling into sub-problems, and 2) both sub-problems can be solved with sufficiently high quality using recent 3D learning algorithms and mesh-based physical simulation techniques. Video presentation available at this link.