Comprehensive Evaluation on Lexical Normalization: Boundary-Aware Approaches for Unsegmented Languages

ACL ARR 2025 May Submission2504 Authors

Abstract: Lexical normalization research has sought to tackle the challenge of processing informal expressions in user-generated text, yet the absence of comprehensive evaluations leaves it unclear which methods excel across multiple perspectives. Focusing on unsegmented languages, we make three key contributions: (1) creating a large-scale, multi-domain Japanese normalization dataset, (2) developing normalization methods based on state-of-the-art pre-trained models, and (3) conducting experiments across multiple evaluation perspectives. Our experiments show that both encoder-only and decoder-only approaches achieve promising results in both accuracy and efficiency.

Paper Type: Long

Research Area: Phonology, Morphology and Word Segmentation

Research Area Keywords: morphological segmentation

Contribution Types: Data resources

Languages Studied: Japanese,Thai

Submission Number: 2504