Gen2Sim: Scaling up Simulation with Generative Models for Robotic Skill Learning
Keywords: Policy Learning in Simulation, Manipulation, Image-to-3D, LLM
TL;DR: Scaling up simulation environments for robotic skill learning using foundational generative models of images and language.
Abstract: We propose Generation to Simulation (Gen2Sim), a method for scaling up robot skill learning in simulation by automatically generating simulation 3D assets, scenes, task definitions, task decompositions and reward functions, capitalizing over large pre-trained generative models of language and images. We propose methods for 3D simulation asset generation from lifting open-world 2D object images using image diffusion models and LLM queries for plausible ranges of physical parameters. We then chain-of-thought prompt LLMs to parse URDF files of generated and human-developed assets to generate task descriptions, task decomposition, and corresponding reward functions, based on the assets and scene affordances. We train reinforcement learning policies in the simulation environments using our generated tasks supervised by the generated reward functions. We demonstrate successful policy learning for a number of long-horizon tasks using Gen2Sim, without any human involvement. Our work contributes hundreds of simulated assets and tasks for articulated and novel 3D object assets, taking a step towards fully autonomous robotic manipulation skill acquisition in simulation.
Submission Number: 40
Loading