Pronunciation-Lexicon Free Training for Phoneme-based Crosslingual ASR via Joint Stochastic Approximation

ACL ARR 2025 February Submission7657 Authors

Abstract:

Recently, pre-trained models with phonetic supervision have demonstrated their advantages for crosslingual speech recognition in data efficiency and information sharing across languages. However, a limitation is that a pronunciation lexicon is needed for such phoneme-based crosslingual speech recognition.
In this study, we aim to eliminate the need for pronunciation lexicons and propose a latent variable model based method, with phonemes being treated as discrete latent variables. The new method consists of a speech-to-phoneme (S2P) model and a phoneme-to-grapheme (P2G) model, and a grapheme-to-phoneme (G2P) model is introduced as an auxiliary inference model.
To jointly train the three models, we utilize the joint stochastic approximation (JSA) algorithm, which is a stochastic extension of the EM (expectation-maximization) algorithm and has demonstrated superior performances particularly in estimating discrete latent variable models.
Based on the Whistle multilingual pre-trained S2P model, crosslingual experiments on Polish (130h) and Indonesian (20h) are conducted.
With only 10 minutes of phoneme supervision, the new method, SPG-JSA, achieves 5\% error rate reductions compared to the best cross-lingual fine-tuning approach using subword or full phoneme supervision.
Furthermore, it is found that in language domain adaptation (i.e., utilizing cross-domain text-only data), SPG-JSA outperforms the standard practice of language model fusion via the auxiliary support of the G2P model by 9\% error rate reductions.

Paper Type: Long

Research Area: Speech Recognition, Text-to-Speech and Spoken Language Understanding

Research Area Keywords: automatic speech recognition

Languages Studied: Polish, Indonesian

Submission Number: 7657