Paper:Multi-Source Diffusion Models for Simultaneous Audio Generation and Separation
Abstract: In this work, we define a diffusion-based generative model capable of both music synthesis and source separation by learning the score of the joint probability density of sources sharing a context. Alongside the classic total inference tasks (i.e. generating a mixture, separating the sources), we also introduce and experiment on the partial inference task of source imputation, where we generate a subset of the sources given the others (e.g., play a piano track that goes well with the drums). Additionally, we introduce a novel inference method for the separation task. We train our model on Slakh2100, a standard dataset for musical source separation, provide qualitative results in the generation settings, and showcase competitive quantitative results in the separation setting. Our method is the first example of a single model that can handle both generation and separation tasks, thus representing a step toward general audio models.
Below are some examples of the tasks we can perform with our approach.
Generation
Here we ask the neural model to randomly generate some new music:
Sample #1
Sample #2
Sample #3
Sample #4
Sample #5
Sample #6
Sample #7
Sample #8
Sample #9
Sample #10
Source Imputation (a.k.a. partial generation)
Given a source stem as input (e.g. drums), the neural model generates accompanying instruments:
Original Source #1
Generated accompaniment #1
Original Source #2
Generated accompaniment #2
Original Source #3
Generated accompaniment #3
Original Source #4
Generated accompaniment #4
Original Source #5
Generated accompaniment #5
Original Source #6
Generated accompaniment #6
Original Source #7
Generated accompaniment #7
Original Source #8
Generated accompaniment #8
Source Separation
Finally, it is possible to use our model to extract single sources from an input mixture: