From MIDI to Motion: Learning to Play the Piano at Scale with Bi-Manual Dexterous Robot Hands

Le Chen, Yi Zhao, Jan Schneider, Quankai Gao, Juho Kannala, Bernhard Schölkopf, Joni Pajarinen, Dieter Büchler

Keywords: Bi-manual dexterous robot hands, dataset for robot piano playing, imitation learning, robot learning at scale

Abstract: It has been a long-standing research goal to endow robot hands with human-level dexterity. Bimanual robot piano playing constitutes a task that combines challenges from dynamic tasks, such as generating fast and precise motions, with slower but contact-rich manipulation problems. Although reinforcement learning-based approaches have shown promising results in single-task performance, these methods struggle in a multi-song setting. Our work aims to close this gap and, thereby, enable imitation learning approaches for robot piano playing at scale. To this end, we introduce the Robot Piano 1 Million (RP1M) dataset, containing bimanual robot piano playing motion data of more than one million trajectories. We formulate finger placements as an optimal transport problem, thus enabling automatic annotation of vast amounts of unlabeled songs. With RP1M, we train a multi-song piano playing policy with imitation learning approaches at scale, leveraging flow matching as the policy representation. Experiments show that our method obtains promising results.

Submission Number: 24