Mitigating the Linguistic Gap with Phonemic Representations for Robust Multilingual Language Understanding

Anonymous

Abstract: Approaches to improving multilingual language understanding often require multiple languages during the training phase, rely on complicated training techniques, and---importantly---struggle with significant performance gaps between high-resource and low-resource languages. We hypothesize that the performance gaps between languages are affected by linguistic gaps between those languages and provide a novel solution for robust multilingual language modeling by employing phonemic representations (specifically, using phonemes as input tokens to LMs rather than subwords). We present quantitative evidence from three cross-lingual tasks that demonstrate the effectiveness of phonemic representation, which is further justified by a theoretical analysis of the cross-lingual performance gap.

Paper Type: short

Research Area: Multilinguality and Language Diversity

Contribution Types: NLP engineering experiment, Approaches to low-resource settings

Languages Studied: English,Swahili,Urdu,French,German,Korean,Ukrainian,Hindi

Preprint Status: We are considering releasing a non-anonymous preprint in the next two months (i.e., during the reviewing process).

A1: yes

A1 Elaboration For Yes Or No: Section 6

A2: n/a

A2 Elaboration For Yes Or No: We have no potential risks are expected.

A3: yes

A3 Elaboration For Yes Or No: Abstract, Section 1

B: yes

B1: yes

B1 Elaboration For Yes Or No: Section 3

B2: yes

B2 Elaboration For Yes Or No: Appendix A

B3: yes

B3 Elaboration For Yes Or No: Section 3.2 Aside from providing license information and its example usage, there was no intended use specified. We utilized the datasets as they were intended to be, for non-commercial research purpose, whether or not it was specified.

B4: n/a

B5: yes

B5 Elaboration For Yes Or No: Appendix A.

B6: yes

B6 Elaboration For Yes Or No: Appendix A.

C: yes

C1: yes

C1 Elaboration For Yes Or No: Appendix A.

C2: yes

C2 Elaboration For Yes Or No: Appendix A.

C3: yes

C3 Elaboration For Yes Or No: Appendix A

C4: yes

C4 Elaboration For Yes Or No: Section A.3

D: no

E: no