XLS-R fine-tuning on noisy word boundaries for unsupervised speech segmentation into words

Robin Jonathan Algayres, Pablo J. Diego Simon, Benoît Sagot, Emmanuel Dupoux

Submission Type: Regular Short Paper

Submission Track: Speech and Multimodality

Keywords: unsupervised speech segmentation into words, self-supervised Learning, self-training

TL;DR: Fine-tuning SSL models with DP-Parse boundaries

Abstract: Due to the absence of explicit word boundaries in the speech stream, the task of segmenting spoken sentences into word units without text supervision is particularly challenging. In this work, we leverage the most recent self-supervised speech models that have proved to quickly adapt to new tasks through fine-tuning, even in low resource conditions. Taking inspiration from semi-supervised learning, we fine-tune an XLS-R model to predict word boundaries themselves produced by top-tier speech segmentation systems: DPDP, VG-HuBERT and DP-Parse. Once XLS-R is fine-tuned, it is used to infer new word boundary labels that are used in turn for another fine-tuning step. Our method consistently improves the performance of each system and set a new state-of-the-art that is, on average 130% higher than the previous one as measured by the F1 score on correctly discovered word tokens on five corpora featuring different languages. Finally, our system can segment speech from languages unseen during fine-tuning in a zero-shot fashion.

Submission Number: 111