JA-TN: Pick-and-Place Towel Shaping from Crumpled States based on TransporterNet with Joint-Probability Action Inference

Halid Abdulrahim Kadi, Kasim Terzić

Keywords: Cloth Manipulation, Imitation Learning, Sim2Real Transfer

TL;DR: Our JA-TN towel-shaping controller extends the TransporterNet architecture with our joint-probability action inference and achieves state-of-the-art performance for flattening, folding-from-flattened and folding-from-crumpled tasks in simulation.

Abstract: Towel manipulation is a crucial step towards more general cloth manipulation. However, folding a towel from an arbitrarily crumpled state and recovering from a failed folding step remain critical challenges in robotics. We propose joint-probability action inference JA-TN, as a way to improve TransporterNet's operational efficiency; to our knowledge, this is the first single data-driven policy to achieve various types of folding from most crumpled states. We present three benchmark domains with a set of shaping tasks and the corresponding oracle policies to facilitate the further development of the field. We also present a simulation-to-reality transfer procedure for vision-based deep learning controllers by processing and augmenting RGB and/or depth images. We also demonstrate JA-TN's ability to integrate with a real camera and a UR3e robot arm, showcasing the method's applicability to real-world tasks.

Student Paper: yes

Submission Number: 239