Improving Chinese Pop Song and Hokkien Gezi Opera Singing Voice Synthesis by Enhancing Local Modeling
Submission Type: Regular Long Paper
Submission Track: Speech and Multimodality
Submission Track 2: Computational Social Science and Cultural Analytics
Keywords: Singing voice synthesis, local modeling enhancement, local adaptive weights loss, Hokkien Gezi Opera, Chinese pop song
TL;DR: To address local incongruity problem in singing voice synthesis task, we propose two methods to enhance local modeling in the acoustic model.
Abstract: Singing Voice Synthesis (SVS) strives to synthesize pleasing vocals based on music scores and lyrics. The current acoustic models based on Transformer usually process the entire sequence globally and use a simple L1 loss. However, this approach overlooks the significance of local modeling within the sequence and the local optimization of the hard-to-synthesize parts in the predicted mel-spectrogram. Consequently, the synthesized audio exhibits local incongruities (\textsl{e.g.}, local pronunciation jitter or local noise). To address this problem, we propose two methods to enhance local modeling in the acoustic model. First, we devise a nearest neighbor local attention, where each phoneme token focuses only on the adjacent phoneme tokens located before and after it. Second, we propose a phoneme-level local adaptive weights loss function that enables the model to focus more on the hard-to-synthesize parts of the mel-spectrogram. We have verified the universality of our methods on public Chinese pop song and Hokkien Gezi Opera datasets. Extensive experiments have demonstrated the effectiveness of our methods, resulting in significant improvements in both objective and subjective evaluations when compared to the strong baselines. Our code and demonstration samples are available at
\href{https://github.com/baipeng1/SVSELM}{https://github.com/baipeng1/SVSELM}.
Submission Number: 475
Loading