SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation

Nadun Ranawaka Arachchige, Josiah Wong, Wei-Lin Pai, Wei-Teng Chu, Tianyuan Dai, Masoud Moghani, Hang Yin, Yunfan Jiang, Wesley Durbano, Brandon Bryce Huynh, Linxi Fan, Danfei Xu, Ruohan Zhang, Li Fei-Fei, Bowen Wen, Ajay Mandlekar, Yuke Zhu

Reviewer: ~Nadun_Ranawaka_Arachchige1, ~Josiah_Wong1

Keywords: real-to-sim, sim-to-real, scene generation, robot manipulation, robot learning

Abstract: Training and evaluating robot policies in the real world can be costly and difficult to scale. Simulation offers a sandbox for efficient policy evaluation and generation of policy training samples, but traditionally requires substantial engineering effort to construct simulated environments that align with reality. While this burden has been alleviated by prior work proposing end-to-end methods for generating pre-processed and annotated ``sim-ready'' scenes useful for downstream robotics tasks, these approaches often cannot be further tuned or controlled by the human operator, limiting the applicability of the generated data to the target tasks. To mitigate these challenges, we introduce a novel modularized and automated system named \textsc{SimFoundry}, enabling zero-shot real-to-sim scene construction from a single image. Our system supports automated object, scene, and task editing, enabling the generation of infinite variations of the real-world scene for training more generalizable policies. We leverage our system in both training (real-to-sim-to-real) and evaluation (real-to-sim) settings and show that our system can generate useful training data, as well as sim-ready scenes that produce correlative signals to real-robot evaluations of trained policies, resulting in more robust co-trained policies across a broad range of robot manipulation tasks, including multiple tasks that surpass the complexity of those shown by prior work and require multiple steps, articulated interaction, and bimanual coordination.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 28