Neural Field Dynamics Model for Granular Object Piles Manipulation

Shangjie Xue, Shuo Cheng, Pujith Kachana, Danfei Xu

Keywords: Deformable Object Manipulation, Manipulation Planning

TL;DR: Our approach combines trajectory optimization and differentiable rendering for granular object manipulation. It introduces a unified density-field-based representation for object states and actions, utilizing a FCN to predict physical dynamics.

Abstract: We present a learning-based dynamics model for granular material manipulation. Drawing inspiration from computer graphics' Eulerian approach, our method adopts a fully convolutional neural network that operates on a density field-based representation of object piles, allowing it to exploit the spatial locality of inter-object interactions through the convolution operations. This approach greatly improves the learning and computation efficiency compared to existing latent or particle-based methods and sidesteps the need for state estimation, making it directly applicable to real-world settings. Furthermore, our differentiable action rendering module makes the model fully differentiable and can be directly integrated with a gradient-based algorithm for curvilinear trajectory optimization. We evaluate our model with a wide array of piles manipulation tasks both in simulation and real-world experiments and demonstrate that it significantly exceeds existing methods in both accuracy and computation efficiency. More details can be found at https://sites.google.com/view/nfd-corl23/

Student First Author: yes

Instructions: I have read the instructions for authors (https://corl2023.org/instructions-for-authors/)

Video: https://sites.google.com/view/nfd-corl23/

Website: https://sites.google.com/view/nfd-corl23/

Code: https://sites.google.com/view/nfd-corl23/