Phys2Real: Physically-Informed Gaussian Splatting for Adaptive Sim-to-Real Transfer in Robotic Manipulation

Maggie Wang, Stephen Tian, Jiajun Wu, Mac Schwager

Keywords: sim-to-real transfer, reinforcement learning, robotic manipulation, digital twins, gaussian splatting, vision-language models

TL;DR: Phys2Real is a real-to-sim-to-real pipeline that improves robotic manipulation by conditioning policies on physical parameters estimated from vision-language models

Abstract: Learning robotic manipulation policies directly in the real world can be expensive and time-consuming, motivating the use of simulation for scalable training. However, effective sim-to-real transfer remains a central challenge in reinforcement learning for robotic manipulation, particularly for tasks that require precise physical dynamics. We present Phys2Real, a real-to-sim-to-real pipeline that generates object-centric digital twins using geometry from 3D reconstructions and physical priors inferred from vision-language models (VLMs). Our approach combines 3D Gaussian Splatting (GSplat) for high-fidelity geometric reconstructions with VLM-based estimates of physical parameters, such as friction and center of mass (CoM). Unlike domain randomization, which trains policies to be robust across broad parameter ranges and often results in averaged behaviors that may not account for object-specific dynamics, Phys2Real conditions reinforcement learning (RL) policies on known physical parameters during training and VLM-inferred parameter estimates at test time. This conditioning enables precise adaptation to novel objects. We evaluate our method on two planar pushing tasks: a T-block with low friction and a hammer with off-center mass distribution, showing improvement in accuracy, success rate (100\% vs 60\%), and task completion time compared to domain-randomization baselines. Phys2Real offers a step toward more adaptable manipulation systems that integrate visual reconstruction, physical reasoning, and adaptive control.

Submission Number: 16