Does Visual Rendering Bypass Tokenization? Investigating Script-Tokenizer Misalignment in Pixel-Based Language Models

ACL ARR 2026 January Submission8402 Authors

Keywords: multilingualism, tokenization, low-resource languages, pixel-based language modeling, indonesian languages

Abstract: While pixel-based language modeling aims to bypass the sub-word tokenization bottleneck by rendering text as images, recent multimodal variants such as DualGPT reintroduce text tokenizers to improve autoregressive performance. We investigate a fundamental question, **does visual rendering truly decouple a model from tokenization constraints?** Focusing on four Indonesian low-resource local languages that have their own non-Latin scripts (i.e., Javanese, Balinese, Sundanese, and Lampungnese), we evaluate the impact of script-tokenizer alignment within the DualGPT architecture. Our results show that, despite visual rendering, reintegrating a text tokenizer into the architecture reintroduces the same issue that pixel-based language modeling aims to resolve, which is the tokenizer misalignment problem. Despite having lower OOV and fertility rates, we show that the Llama 2 tokenizer performs significantly worse than a custom tokenizer, with improvements of up to +30.15 chrF++. Our findings serve as a warning for future multimodal variants, as text tokenizers remain a significant barrier to equitable models.

Paper Type: Short

Research Area: Multilinguality and Language Diversity

Research Area Keywords: multilingualism,less-resourced languages,multilingual evaluation

Contribution Types: Approaches to low-resource settings

Languages Studied: Javanese,Balinese,Sundanese,Lampung

Submission Number: 8402