Toward Speech Articulation Detection through Smartphone

Aslan B. Wong, Xia Chen, Hao Ren, Usman Saleh Toro, Kaishun Wu

Abstract: Articulation problems seriously impact speech communication and comprehension. Hence, Automatic Speech Recognition (ASR) has been applied to detect and analyze articulation from speech signals in various applications such as: clinical protocols, foreign language learning, and language proficiency testing. However, articulation detection and analysis has not been adequately evaluated due to its complex nature. The challenging task is that speech signal alone contains insufficient information for articulation detection and analysis. Hence, we propose an alternative approach for articulation detection and analysis by developing a system that senses users’ articulatory organs (tongue and lips) based on phonetic rules. The system employs speech and ultrasonic signals simultaneously to read lip shape and tongue position. We also implemented the proposed technique on an off-the-shelf smartphone to enhance applicability in real-world scenarios. We evaluated the system using four languages: French, Japanese, Korean, and Mandarin Chinese. The result of our evaluation shows that our system is robust in recognizing vowel sound articulation with an overall accuracy of 94.74%.