Lucid-XR: An Extended-Reality Data Engine for Robotic Manipulation

Yajvan Ravan, Adam Rashid, Alan Yu, Kai McClennen, Gio Huh, Kevin Yang, Zhutian Yang, Qinxi Yu, Xiaolong Wang, Phillip Isola, Ge Yang

Keywords: mixed reality, extended reality, robot manipulation, simulation, mujoco

TL;DR: We present a scalable recipe for creating synthetic robot training data via generative AI models and web-based physics simulation in extended reality.

Abstract: We introduce Lucid-XR, a generative data engine for creating diverse and realistic-looking data to train real-world robot systems. At the core of Lucid-XR is vuer, a web-based physics simulation environment that runs directly on the XR headset, enabling internet-scale access to immersive, latency-free virtual interactions without requiring specialized equipment. The complete system integrates on-device physics simulation with on-device human-to-robot pose retargeting, that are further amplified by a physics-guided video generation pipeline commandable with natural language specifications. We demonstrate zero-shot sim-to-real transfer of robot visual policies, trained entirely on Lucid-XR's synthetic data, across bimanual and dexterous manipulation tasks that involve flexible materials, adhesive interaction between particles, and rigid body contact.

Submission Number: 858