Tokenization Multiplicity Leads to Arbitrary Price Variation in LLM-as-a-service

Ivi Chatzi, Nina Corvelo Benz, Stratis Tsirtsis, Manuel Gomez Rodriguez

Keywords: LLM-as-a-service, non-canonical tokenization, constrained generation

Domains: Language and Learning

TL;DR: LLMs can generate the same output string under different tokenizations which leads to price variation in pay-per-token LLM-as-a-service APIs

External Link: https://arxiv.org/abs/2506.06446

Abstract: Providers of LLM-as-a-service have predominantly adopted a simple pricing model: users pay a fixed price per token. Consequently, one may think that the price two different users would pay for the same output string under the same input prompt is the same. In our work, we show that, surprisingly, this is not (always) true. We find empirical evidence that, particularly for non-english outputs, both proprietary and open-weights LLMs often generate the same (output) string with multiple different tokenizations, even under the same input prompt, and this in turn leads to arbitrary price variation. To address the problem of tokenization multiplicity, we introduce canonical generation, a type of constrained generation that restricts LLMs to only generate canonical tokenizations---the unique tokenization in which each string is tokenized during the training process of an LLM. Further, we introduce an efficient sampling algorithm for canonical generation based on the Gumbel-Max trick. Experiments on a variety of natural language tasks demonstrate that canonical generation is comparable to standard generation in terms of performance and runtime, and it solves the problem of tokenization multiplicity.

Submission Number: 98