LEGATO: Large-scale End-to-end Generalizable Approach to Typeset OMR

Guang Yang, Victoria Ebert, Nazif Can Tamer, Brian Siyuan Zheng, Luiza Amador Pozzobon, Noah A. Smith

Keywords: Optical Music Recognition, AI for Music, Multimodal Learning

TL;DR: Legato is the first large-scale pretrained OMR model that achieves the state of the art on various datasets.

Abstract: We propose Legato, a new end-to-end model for optical music recognition (OMR), a task of converting music score images to machine-readable documents. Legato is the first large-scale pretrained OMR model capable of recognizing full-page or multi-page typeset music scores and the first to generate documents in ABC notation, a concise, human-readable format for symbolic music. Bringing together a pretrained vision encoder with an ABC decoder trained on a dataset of more than 214K images, our model exhibits the strong ability to generalize across various typeset scores. We conduct comprehensive experiments on a range of datasets and metrics and demonstrate that Legato outperforms the previous state of the art. On our most realistic dataset, we see a 68\% and 47.6\% absolute error reduction on the standard metrics TEDn and OMR-NED, respectively.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 14007