Retrieval Dexterity: Efficient Object Retrieval in Clutters with Dexterous Hand

Tawei Chou, Fengshuo Bai, Yaodong Yang

Abstract: Retrieving objects buried beneath multiple others is not only challenging but also time-consuming. Performing manipulation in such environments presents significant difficulty due to complex contact relationships. Existing methods typically address this task by sequentially grasping and removing each occluding object, resulting in lengthy execution times and requiring impractical grasping capabilities for every occluding object. In this paper, we present a dexterous arm-hand system for efficient object retrieval in cluttered environments. Our approach leverages large-scale parallel reinforcement learning (RL) within diverse and carefully designed cluttered environments to train policies. To further enhance policy performance, we introduce a spatial-aware representation module that encodes the occlusion and spatial relationships among the target object, robotic hand, and surrounding clutter. This structured representation guides the policy to exhibit emergent manipulation skills (e.g., pushing, stirring, and poking) that effectively clear occluding objects to expose sufficient surface area of the target. We conduct extensive evaluations across 16 household objects in diverse clutter configurations, demonstrating superior retrieval performance and efficiency on both trained and unseen objects. Furthermore, we successfully transfer the learned policies to a real-world dexterous multi-fingered robot system, validating their practical applicability. Videos can be found on our project website: https://RetrDex.github.io.