Learning Generalizable Dexterous Manipulation from Human Grasp Affordance

Yueh-Hua Wu, Jiashun Wang, Xiaolong Wang

Keywords: Generalized policy learning, Dexterous manipulation, Affordance model, Reinforcement learning, Imitation learning

TL;DR: We propose a pipeline that generates large-scale demonstrations for dexterous manipulation and an imitation learning algorithm that generalizes to novel objects absent during training.

Abstract: Dexterous manipulation with a multi-finger hand is one of the most challenging problems in robotics. While recent progress in imitation learning has largely improved the sample efficiency compared to Reinforcement Learning, the learned policy can hardly generalize to manipulate novel objects, given limited expert demonstrations. In this paper, we propose to learn dexterous manipulation using large-scale demonstrations with diverse 3D objects in a category, which are generated from a human grasp affordance model. This generalizes the policy to novel object instances within the same category. To train the policy, we propose a novel imitation learning objective jointly with a geometric representation learning objective using our demonstrations. By experimenting with relocating diverse objects in simulation, we show that our approach outperforms baselines with a large margin when manipulating novel objects. We also ablate the importance of 3D object representation learning for manipulation. We include videos and code on the project website: https://kristery.github.io/ILAD/ .

Student First Author: yes

Website: https://kristery.github.io/ILAD/

Code: https://github.com/kristery/dex-affordance

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:2204.02320/code)