Pronunciation Training through Sensing of Tongue and Lip Motion via Smartphone

Aslan B. Wong, Kaishun Wu

Abstract: Learning a foreign language pronunciation is the most challenging task for non-native speakers. Compared to scoring feedback on pronunciation assessment, informative feedback for a self-correct pronunciation training system is still left behind. To bring this to an end, we propose a pronunciation training system that can give articulation feedback to a user. Our approach focuses on correcting the user's articulation. We employ simultaneously two ranges of acoustic signals, both speech and ultrasonic signal, to recognize lip shape and tongue position. The proposed technique is also implemented into an off-the-shelf smartphone to be more accessible. Our system is currently working on 30 vowels in four languages: French, Japanese, Korean, and Mandarin Chinese.