Dexterous Non-prehensile Manipulation using Environment Guided Diffusion Models

Shaoxiong Yao, Chengyi Xing, Yuki Shirai, Devesh K. Jha, Diego Romeres

Keywords: Non-prehensile manipulation, diffusion model

TL;DR: We develop a guided diffusion framework for generating environment-conditioned non-prehensile manipulation actions.

Abstract: Humans naturally use non-prehensile actions, such as sliding and poking, to manipulate objects that are not immediately graspable. Enabling robots with a similar capability could significantly broaden the range of environments in which they can operate effectively. In this work, we propose a novel framework for generating non-prehensile manipulation actions conditioned on both object and environmental geometry. Our method builds on a guided diffusion formulation that adapts manipulation behaviors to the environment by encoding object and environment contacts as differentiable loss terms during training-free sampling. To train and evaluate this approach, we develop a simulation-based pipeline for collecting diverse manipulation behaviors across objects with different geometries and contact conditions. Experiments in several challenging environments show that the learned diffusion model adapts effectively to environmental context and achieves around 2$\times$ higher success rates compared to baseline approach.

Submission Number: 21