Leveraging 3D Reconstruction for Mechanical Search on Cluttered Shelves

Seungyeon Kim, Young Hun Kim, Yonghyeon Lee, Frank C. Park

Keywords: Mechanical search, Object rearrangement, Prehensile and Non-prehensile manipulation

TL;DR: We propose a 3D reconstruction-based meachnical search algorithm on cluttered shelves.

Abstract: Finding and grasping a target object on a cluttered shelf, especially when the target is occluded by other unknown objects and initially invisible, remains a significant challenge in robotic manipulation. While there have been advances in finding the target object by rearranging surrounding objects using specialized tools, developing algorithms that work with standard robot grippers remains an unresolved issue. In this paper, we introduce a novel framework for finding and grasping the target object using a standard gripper, employing pushing and pick and-place actions. To achieve this, we introduce two indicator functions: (i) an existence function, determining the potential presence of the target, and (ii) a graspability function, assessing the feasibility of grasping the identified target. We then formulate a model-based optimal control problem. The core component of our approach involves leveraging a 3D recognition model, enabling efficient estimation of the proposed indicator functions and their associated dynamics models. Our method succeeds in finding and grasping the target object using a standard robot gripper in both simulations and real-world settings. In particular, we demonstrate the adaptability and robustness of our method in the presence of noise in real-world vision sensor data. The code for our framework is available at https://github.com/seungyeon-k/Search-for-Grasp-public.

Student First Author: yes

Instructions: I have read the instructions for authors (https://corl2023.org/instructions-for-authors/)

Video: https://www.youtube.com/watch?v=FoejNGHf1XM&t=2s

Code: https://github.com/seungyeon-k/Search-for-Grasp-public