Stop Taking Tokenizers for Granted: They Are Core Design Decisions in Large Language Models

Open Webpage

Sawsan Alqahtani, Mir Tafseer Nayeem, Md. Tahmid Rahman Laskar, Tasnim Mohiuddin, M. Saiful Bari

Published: 2026, Last Modified: 28 May 2026CoRR 2026EveryoneRevisionsBibTeXCC BY-SA 4.0