Code-Switched Language Identification is Harder Than You Think

Anonymous

Abstract: Code switching (CS) is a very common phenomenon in written and spoken communication, but is handled poorly by many NLP applications. Looking to the application of building CS corpora, we explore CS language identification for corpus building. We make the task more realistic by scaling it to more languages and considering models with simpler architectures for faster inference. We also reformulate the task as a sentence-level multi-label tagging problem to make it more tractable. Having defined the task, we investigate three reasonable architectures for this task and define metrics which better reflect desired performance. We present empirical evidence that no current approach is adequate, and finally provide recommendations for future work in this area.

Paper Type: long

Research Area: Multilinguality and Language Diversity

Contribution Types: Approaches to low-resource settings, Data analysis, Position papers

Languages Studied: English, Spanish, Basque, Chinese, Turkish, Indonesian, Modern Standard Arabic, Egyptian Arabic, plus c. 200 languages covered by FLORES-200

Consent To Share Submission Details: On behalf of all authors, we agree to the terms above to share our submission details.