Synthetic contact maps to predict grasp regions on objects

Pablo Martinez-Gonzalez, David Mulero-Perez, Sergiu Oprea, Manuel Benavent-Lledó, Sergio Orts-Escolano, José García Rodríguez

Abstract: From early ages, we learn to naturally grasp objects from our trial-and-error interaction with the environment. We end up excelling at manipulating objects, such that grabbing a pan by its handle is all but unconscious. However, performing the same task from a machine's perspective is particularly challenging, as it requires an in-depth understanding of object manipulation. We could approximate such an understanding by capturing the hand-object contact resulting from human grasps, i.e., contact maps. This has already been done using cumbersome capture systems, and not on a large scale. In this work, we simplify and accelerate such a process by generating contact maps from our interaction with household objects in photorealistic virtual reality environments. To the best of our knowledge, we are the first to generate contact maps at scale from our interaction in a virtual scenario. We train an image-to-image translation method to predict grasp regions on objects, demonstrating the usefulness of our generated contact maps. We provide all the necessary tools, code, and dataset to foster applications where hand-object understanding is necessary.