Structured deep generative models for sampling on constraint manifolds in sequential manipulation

Joaquim Ortiz-Haro, Jung-Su Ha, Danny Driess, Marc Toussaint

Keywords: Generative Models, Nonlinear Optimization, Constraint Graph, Robotic Sequential Manipulation

Abstract: Sampling efficiently on constraint manifolds is a core problem in robotics. We propose Deep Generative Constraint Sampling (DGCS), which combines a deep generative model for sampling close to a constraint manifold with nonlinear constrained optimization to project to the constraint manifold. The generative model is conditioned on the problem instance, taking a scene image as input, and it is trained with a dataset of solutions and a novel analytic constraint term. To further improve the precision and diversity of samples, we extend the approach to exploit a factorization of the constrained problem. We evaluate our approach in two problems of robotic sequential manipulation in cluttered environments. Experimental results demonstrate that our deep generative model produces diverse and precise samples and outperforms heuristic warmstart initialization.