End-to-End ASR: from Supervised to Semi-Supervised Learning with Modern ArchitecturesDownload PDF

Published: 02 Jul 2020, Last Modified: 22 Oct 2023SAS 2020Readers: Everyone
Keywords: speech recognition, pseudo-labelling
TL;DR: Study of pseudo-labeling for the semi-supervised ASR training of different architectures with either CTC or Seq2Seq loss functions and 60k hours unlabeled speech.
Abstract: We study pseudo-labeling for the semi-supervised training of ResNet, Time-Depth Separable ConvNets, and Transformers for speech recognition, with either CTC or Seq2Seq loss functions. We perform experiments on the standard Librispeech dataset, and leverage additional unlabeled data from Librivox through pseudo-labeling. We show that while Transformer-based acoustic models have superior performance with the supervised dataset alone, semi-supervision improves all models across architectures and loss functions and bridges much of the performance gaps between them. In doing so, we reach a new state-of-the-art for end-to-end acoustic models decoded with an external language model in the standard supervised learning setting, and a new absolute state-of-the-art with semi-supervised training. Finally, we study the effect of leveraging different amounts of unlabeled audio, propose several ways of evaluating the characteristics of unlabeled audio which improve acoustic modeling, and show that acoustic models trained with more audio rely less on external language models.
Double Submission: No
Community Implementations: [![CatalyzeX](/images/catalyzex_icon.svg) 1 code implementation](https://www.catalyzex.com/paper/arxiv:1911.08460/code)
7 Replies