Speech Recognition and Meaning Interpretation: Towards Disambiguation of Structurally Ambiguous Spoken Utterances in Indonesian

Ruhiyah Faradishi Widiaputri, Ayu Purwarianti, Dessi Puji Lestari, Kurniawati Azizah, Dipta Tanaya, Sakriani Sakti

Submission Type: Regular Long Paper

Submission Track: Speech and Multimodality

Submission Track 2: Resources and Evaluation

Keywords: structural ambiguity in sentences, prosodic information, speech recognition, speech-to-text mapping, meaning interpretation

TL;DR: Indonesian ASR and meaning interpretation of structurally ambiguous utterances

Abstract: Despite being the world's fourth-most populous country, the development of spoken language technologies in Indonesia still needs improvement. Most automatic speech recognition (ASR) systems that have been developed are still limited to transcribing the exact word-by-word, which, in many cases, consists of ambiguous sentences. In fact, speakers use prosodic characteristics of speech to convey different interpretations, which, unfortunately, these systems often ignore. In this study, we attempt to resolve structurally ambiguous utterances into unambiguous texts in Indonesian using prosodic information. To the best of our knowledge, this might be the first study to address this problem in the ASR context. Our contributions include (1) collecting the Indonesian speech corpus on structurally ambiguous sentences\footnote{Our corpus is available at \url{https://github.com/ha3ci-lab/struct_amb_ind}}; (2) conducting a survey on how people disambiguate structurally ambiguous sentences presented in both text and speech forms; and (3) constructing an Indonesian ASR and meaning interpretation system by utilizing both cascade and direct approaches to map speech to text, along with two additional prosodic information signals (pause and pitch). The experimental results reveal that it is possible to disambiguate these utterances. In this study, the proposed cascade system, utilizing Mel-spectrograms concatenated with F0 and energy as input, achieved a disambiguation accuracy of 79.6\%, while the proposed direct system with the same input yielded an even more impressive disambiguation accuracy of 82.2\%.

Submission Number: 5641