MUTANT: A Recipe for Multilingual Tokenizer Design

ACL ARR 2026 January Submission6451 Authors

Keywords: Tokenizer, Multilingual LLMs, Multilingual Tokenizer, Indic, Tokenization Efficiency, LLMs, Indic Tokenizer, Indian Languages, BPE

Abstract:

Tokenizers play a crucial role in determining the performance, training efficiency, and the inference cost of Large Language Models (LLMs). Designing effective tokenizers for multilingual LLMs is particularly challenging due to diverse scripts and rich morphological variation. While subword methods like Byte Pair Encoding (BPE) are widely adopted, their effectiveness in multilingual settings remains underexplored.
We present MUTANT, a recipe for building multilingual tokenizers, with careful vocabulary and training data design, language-aware pre-tokenization, and subword and multiword aware training. We also introduce MUTANT-Indic, a tokenizer for India-specific multilingual LLMs, that produces linguistically coherent tokens and achieves state-of-the-art performance. Evaluated across English, $22$ Indian languages and code data, our tokenizer improves the average fertility score by 39.5% over LLaMA4 and by 18% over Sutra (the current best). This translates to 44% improvement in inference throughput over LLaMA4 while maintaining comparable performance on English and Indic benchmarks. We present detailed ablations across tokenizer training data size, vocabulary size, merging techniques, and pre-tokenization strategies, demonstrating the robustness of our design choices.

Paper Type: Long

Research Area: Multilinguality and Language Diversity

Research Area Keywords: Efficient/Low-Resource Methods for NLP, Language Modeling, Multilingualism and Cross-Lingual NLP

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Approaches to low-resource settings, Approaches low compute settings-efficiency

Languages Studied: Hindi, Marathi, Maithili, Dogri, Konkani, Sanskrit, Nepali, Kashmiri, Assamese, Bengali, Punjabi, Urdu, Sindhi, Kannada, Malayalam, Tamil, Telugu, Manipuri, Santali, Bodo

Submission Number: 6451