Pronunciation-Lexicon Free Training for Phoneme-based Crosslingual ASR via Joint Stochastic Approximation

Saierdaer Yusuyin, Te Ma, Hao Huang, Zhijian Ou

Keywords: speech recognition, crosslingual, joint stochastic approximation, phoneme;

Abstract:

Recently, pre-trained models with phonetic supervision have demonstrated their advantages for crosslingual speech recognition in data efficiency and information sharing across languages. The Whistle approach relaxes the requirement of gold-standard human-validated phonetic transcripts and adopts weakly-phonetic supervision; however, a limitation is that a pronunciation lexicon is needed for such phoneme-based crosslingual speech recognition.
In this study, we aim to eliminate the need for the pronunciation lexicon and propose a latent variable model based method, with phonemes being treated as discrete latent variables. The new method consists of a speech-to-phoneme (S2P) model and a phoneme-to-grapheme (P2G) model, and a grapheme-to-phoneme (G2P) model is introduced as an auxiliary inference model.
To jointly train the three models, we utilize the joint stochastic approximation (JSA) algorithm, which is a stochastic extension of the EM (expectation-maximization) algorithm and has demonstrated superior performances particularly in estimating discrete latent variable models.
Based on the Whistle multilingual pre-trained S2P model, crosslingual experiments on Polish (130h) and Indonesian (20h) are conducted.
By using only 10 minutes of phoneme supervision, the new method, called as Whistle-JSA, performs close to crosslingual fine-tuning with the full set of phoneme supervision, and on par with the method of crosslingual fine-tuning with subword supervision.
Furthermore, it is found that in language domain adaptation (i.e., utilizing cross-domain text-only data), Whistle-JSA outperforms the standard practice of language model fusion via the auxiliary support of the G2P model.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 9206