Learning Neural Implicit Functions as Object Representations for Robotic Manipulation

Jung-Su Ha, Danny Driess, Marc Toussaint

Keywords: Representation Learning, nerual implicit representation, robotic manipulation, task and motion planning

Abstract: Robotic manipulation planning is the problem of finding a sequence of robot configurations that involves interactions with objects in the scene, e.g., grasp, placement, tool-use, etc. To achieve such interactions, traditional approaches require hand-designed features and object representations, and it still remains an open question how to describe such interactions with arbitrary objects in a flexible and efficient way. Inspired by neural implicit representations in 3D modeling, e.g. NeRF, we propose a method to represent objects as neural implicit functions upon which we can define and jointly train interaction features. The proposed pixel-aligned representation is directly inferred from camera images with known camera geometry, naturally acting as a perception component in the whole manipulation pipeline, while at the same time enabling sequential robot manipulation planning.

One-sentence Summary: This work transfers the idea of neural implicit representations to robotic manipulation applications.