Study: The same text costs far more tokens in some languages than in English across AI models

Using AI in languages other than English can cost more, Rest of World reported on October 26. The report is based on “The Tokenizer Tax,” a study the Coalition for Language Equity in AI Research (CLEAR) published on September 29. Models split text into chunks called tokens, and providers charge per token regardless of language. Languages with less digital data get split into smaller fragments, which raises the token count. The researchers call this gap a “tokenizer tax.” More tokens also mean higher latency and less effective room in the context window.

CLEAR tokenized FLORES-200, a set of sentences professionally translated into about 200 languages, with tokenizers from every major lab. Its headline metric is how many times more tokens the costliest language needs than the cheapest. For the 20 languages most common on the web, the ratio sits between 1.5x and 3.1x with one exception. Across all FLORES-200 languages it ranges from 2.7x for the most equitable tokenizer to 17x in the worst case. Under CLEAR’s grading, no general-purpose frontier model earns better than a B. Google’s Gemini and Gemma family gets a B at 5.1x, Anthropic’s Claude Fable 5 and Opus 4.8 a D at 9.3x, and OpenAI’s GPT-5 an F at 13.8x. The hardest-hit languages use rarely seen scripts, such as Burmese, Tibetan, Khmer and Lao.

Turkish isn’t discussed in the study’s write-up, but it is part of FLORES-200. Per the raw counts CLEAR published on GitHub, the same Turkish text takes roughly 1.3 to 2 times as many tokens as English, depending on the model. The ratio is 1.30 for Llama 4 Scout, 1.34 for Gemini and Gemma, 1.42 for GPT-5, 1.69 for Claude Fable 5 and 2.02 for DeepSeek V4.1 Flash. These ratios aren’t stated in the study; they were calculated from its published counts. CLEAR asks labs to report such ratios in model cards and to move to character- or language-normalized pricing. Dheemanth Reddy, CEO of Maya Research, told Rest of World that falling token prices will ease the cost impact, but that fixing the gap requires better tokenizer design.