Generative Factor Chaining: Coordinated Manipulation with Diffusion-based Factor Graph

Utkarsh Aashu Mishra, Yongxin Chen, Danfei Xu

Keywords: Task and Motion Planning, Diffusion Models, Manipulation Planning

TL;DR: We present Generative Factor Chaining (GFC), a novel approach based on modularized generative models for learning and composing skills in complex tasks.

Abstract: In the realm of challenging long-horizon planning tasks involving multiple manipulators, existing methods encounter computational scalability issues or require an impractical amount of training data. To address these limitations, we present Generative Factor Chaining (GFC), a novel approach based on modularized generative models for learning and composing skills in complex tasks. Our proposed method treats a long-horizon planning task in a complex scene as a spatial-temporal factor graph, where nodes represent objects in the scene and factors denote constraints/skills that connect different objects. By employing the diffusion model framework, different factors can be jointly learned using individual skill data, which is readily obtainable. During inference, these factors can be flexibly composed, possibly with additional constraints, to achieve long-horizon planning. The modular design of GFC enables generalization to unseen planning tasks. We showcase the advantages of our method through real-world experiments. More details can be found at: https://sites.google.com/view/generative-factor-chaining

Submission Number: 2