Autonomous Clay Sculpting from Human Demonstrations with Point Cloud Goal Conditioned Diffusion Policy

Alison Bartsch, Arvind Car, Charlotte Avra, Amir Barati Farimani

Keywords: 3D deformable objects, 3D diffusion policy, point clouds, sculpting, imitation learning

TL;DR: Present a 3D diffusion policy framework for sculpting 3D deformable objects.

Abstract: 3D deformable object manipulation remains a challenge due to the difficulties of state estimation, long-horizon planning, and predicting how the object will deform given an interaction. In this work, we propose SculptDiff, a goal-conditioned diffusion-based imitation learning framework that works with point cloud states to directly learn clay sculpting policies for a variety of target shapes. To the best of our knowledge this is the first real-world end-to-end policy for 3D deformable object manipulation. For sculpting videos, see the project website: https://sites.google.com/andrew.cmu.edu/imitation-sculpting/home.

Submission Number: 15