RoboCook: Long-Horizon Elasto-Plastic Object Manipulation with Diverse ToolsDownload PDF

Published: 30 Aug 2023, Last Modified: 17 Oct 2023CoRL 2023 OralReaders: Everyone
Keywords: Deformable Object Manipulation, Long-horizon Planning, Model Learning, Tool Usage
TL;DR: RoboCook perceives, models, and manipulates elasto-plastic objects for long-horizon tasks like making dumplings and alphabet letter cookies.
Abstract: Humans excel in complex long-horizon soft body manipulation tasks via flexible tool use: bread baking requires a knife to slice the dough and a rolling pin to flatten it. Often regarded as a hallmark of human cognition, tool use in autonomous robots remains limited due to challenges in understanding tool-object interactions. Here we develop an intelligent robotic system, RoboCook, which perceives, models, and manipulates elasto-plastic objects with various tools. RoboCook uses point cloud scene representations, models tool-object interactions with Graph Neural Networks (GNNs), and combines tool classification with self-supervised policy learning to devise manipulation plans. We demonstrate that from just 20 minutes of real-world interaction data per tool, a general-purpose robot arm can learn complex long-horizon soft object manipulation tasks, such as making dumplings and alphabet letter cookies. Extensive evaluations show that RoboCook substantially outperforms state-of-the-art approaches, exhibits robustness against severe external disturbances, and demonstrates adaptability to different materials.
Student First Author: yes
Supplementary Material: zip
Instructions: I have read the instructions for authors (
Publication Agreement: pdf
12 Replies