Probing the Hidden Layers of a Music Generating Language Model
Keywords: machine learning, language models, music generation, probes
TL;DR: We ran a probe experiment on a music generation language model and unveiled a bias in the importance given to input tokens
Abstract: Our study delves into FIGARO, a Transformer-based music generation model, revealing its uneven handling of musical elements like chords and velocity. By running a probe experiment on the model's encoder, we uncover a significant bias in processing certain features, directly impacting music generation. An attention visualization further highlights a distinct pattern correlating with musical bars, offering novel insights into music generation with language models.
Submission Number: 220
Loading