MuseCoco: Generating Symbolic Music from Text

Peiling Lu, Xin Xu, Chenfei Kang, Botao Yu, Chengyi Xing, Xu Tan, Jiang Bian

Primary Area: representation learning for computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: music generation, text to music generation, symbolic music, attribute, controlled

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: In this paper, we propose MuseCoco, which generates symbolic music from text descriptions with musical attributes as the bridge to break down the task into text-to-attribute understanding and attribute-to-music generation stages.

Abstract: Due to the inherent ease of textual input for user engagement, it is natural to generate music from text. In this paper, we introduce MuseCoco (Music Composition Copilot), a system meticulously designed to compose symbolic music from text descriptions. It operates by utilizing musical attributes as a bridge and dividing the process into text-to-attribute understanding stage and attribute-to-music generation stage, which bestows three key advantages: First, it eliminates the need for paired text-to-music data, as text-attribute pairs for the text-to-attribute understanding stage can be automatically synthesized as many as needed, and attributes required for the attribute-to-music generation stage can be directly extracted from music data, thus alleviating the labor-intense process of human annotation. Second, thanks to the explicit attribute design, the system excels in offering precise control over musical attributes, which ensures a high degree of accuracy in shaping the musical output according to user's intentions. Third, it can offer an additional attribute-conditioned control option beyond textual input, enhancing its versatility and usability. Our experimental results demonstrate that MuseCoco significantly outperforms our top-performing baseline model, GPT-4, on musicality, controllability, and overall score, by 45.5\%, 35.2\%, and 47.0\%, respectively. There is also a notable enhancement of approximately 20\% in objective control accuracy. Additionally, we have developed a large-scale model with 1.2 billion parameters, showcasing exceptional controllability and musicality. In practical applications, MuseCoco can serve as a user-friendly tool for musicians, enabling them to effortlessly generate music by simply providing text descriptions, and offering a substantial enhancement in efficiency compared to manually composing music from scratch. Music samples generated by MuseCoco are available via this link https://musecoco.github.io.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 5377