Object-centric Manipulation in Dynamic Environments using Diffused Orientation Fields

Cem Bilaloglu, Tobias Löw, Sylvain Calinon

Keywords: robotics, manipulation, spectral geometry

Abstract: Human environments are filled with objects that exhibit substantial shape variation even within the same category, such as fruits or kitchen utensils. In dynamic settings, a robot will often encounter the same task on the same class of objects with different shapes. For object-centric manipulation skills that require continuous physical interaction along the surface, such as slicing, peeling, or cleaning, this mismatch between the actual and expected shape can lead to failure. To address this, we introduce an object-centric approach that expresses actions in local reference frames adapted to object geometry, enabling shape-invariant task descriptions. Policies represented in these local frames become both robust and adaptable across object instances. We construct the local frames as a smooth orientation field, computed online from raw point clouds and a few keypoints using diffusion processes governed by partial differential equations. We demonstrate that this approach improves transfer of contact-rich tasks such as slicing and peeling across varied objects, enhances robustness under occlusions and keypoint noise, and integrates seamlessly with diverse control paradigms, making it suitable for dynamic, real-world environments.

Submission Number: 11