Tokenization Tax Report 2026

TL;DR

We compared the tokenizers of current LLMs across different texts and languages.

• Strong across the board: Cohere Command A+, Meta’s Muse (Glimmer and Spark; based on Llama 4’s vocab), xAI Grok 4.6, Alibaba Qwen 3.8, OpenAI o200k (GPT-6, GPT 5, gpt-oss)
• Beware of heavy Claude taxes: +65% tokens on average, and +101% on German text

1. Introduction

Large language models (LLMs) process and generate tokens. What exactly makes a token is defined by a model’s tokenizer, or more specifically, its tokenizer’s vocabulary. Each token is an entry in a vocabulary that is usually fixed before training begins, although sometimes additional special tokens are added during post-training.

In most current model tokenizers, the vocabulary is learned from (a subset of) its training dataset via byte pair encoding (BPE). Tokenizer creation starts from individual bytes. The most frequent adjacent byte-pairs are repeatedly merged into new vocabulary entries, stopping once the vocabulary reaches a target size. It produces two outputs: the vocabulary itself, and an ordered list of merges that reproduces the segmentation. Because the training datasets and the target size differ between models, so do the resulting vocabularies. The same text can require substantially different numbers of tokens depending on which model reads or generates it. In case you are looking to brush up your knowledge on tokenization, we recommend C2S4-Tokenization and C6S5-BPE of the Hugging Face LLM course.

Here is an example of two tokenizers processing the same sentence:

When using LLM APIs, you are pretty much always billed per token, with input and output tokens priced separately. This means the number of tokens a text is split into directly influences how much you are charged. But the pricing is not the only thing affected by tokenization efficiency. When comparing models, you should at least have a rough idea on how tokenization influences:

• Price. At a fixed rate per token, a text that takes 30% more tokens costs 30% more to send and 30% more to generate. Depending on tokenization efficiency, price-per-token rates might not be directly comparable.

• Inference speed. At the same decoding throughput in tokens per second, the same response takes 30% longer to generate if its tokenizer needs 30% more tokens. Even with the same model architecture, throughput can change with vocabulary size. Tokenization efficiency therefore matters when comparing tokens-per-second (TPS) rates and tokens-per-minute (TPM) limits.

• Maximum context window. Context limits are specified in tokens. The same window fits different amounts of text depending on the tokenizer. Depending on tokenization efficiency, context window sizes might not be directly comparable.

• Output and reasoning budgets. Limits on generated output are set in tokens, including the reasoning budgets of thinking models. A model that spends more tokens to express a thought might get less reasoning done before it hits the cap. Depending on tokenization efficiency, budget settings might not be directly comparable.

In this post, we measure how large the differences in tokenizer efficiency actually are. We test 20 tokenizers of current proprietary and open-weight models on a dataset covering 21 different kinds of text and prompts. We repeat the same test in 12 languages to examine how tokenizer efficiency varies for different languages.

We are looking for answers to the following questions:

• How many tokens does the same text take across current tokenizers? And how much does this vary by type of text and language?
• Which languages are different model providers optimizing for? What is inside the tokenizer vocabularies?
• How long do tokenizers stay in service across model generations, and how have vocabulary sizes changed over time?

2. Experimental Setup

2.1 Tokenizers

We test 20 tokenizers used by current proprietary and open-weight models, with a wide range of vocabulary sizes from 16k to 262k. Some model developers adopt tokenizers from others, for example, Thinking Machines’ Inkling model re-uses OpenAI’s o200k tiktoken tokenizer. Similarly, Mistral’s Tekken tokenizer is used in NVIDIA’s Nemotron 3 and in Apertus, the Swiss national model, with slight variations in special tokens introduced during post-training. Meta’s Muse Glimmer and Spark use a near-identical variant of Llama 4’s vocabulary. Of the entries in the public Muse Glimmer vocabulary, 99.0% also appear in Llama 4.

2.2 Datasets

Our tests involve a wide variety of texts, differing in sources, register/type and language, as well as code. The first part of our dataset is based on real, human-written documents. Sources include the web, PDFs, encyclopedias, public tenders, news, court, parliament records and open science datasets. The second part contains 3,261 synthetic items covering thirteen assistant-style tasks. For ten of these tasks, we embedded snippets from real documents into stereotypical prompts, reflecting real-world user prompts an LLM-assistant could receive in production. We replicate the dataset in 12 languages: English, German, French, Spanish, Italian, Dutch, Polish, Swedish, Chinese, Japanese, Korean and Arabic, giving us 96 document splits: eight text types in each language.

The ten document-grounded tasks include summarization, extraction to JSON, rewriting, classification, critique, document Q&A, drafting a reply, explaining simply, multi-turn chat and converting key points into a table format.

A third part covers source code. Python, JavaScript, C++, Bash and SQL, which we take from the Stack-Edu dataset. We do not include code in cross-language comparisons and rather treat it as a language of its own. We also drop chunks whose text contains more than 5% non-ASCII chars, so that our comparison actually measures source code rather than the language its comments or docstrings happen to be written in.

2.3 How We Measure

We split real documents and code into chunks targeting 1,000 o200k reference tokens, allowing chunks to span document boundaries. Synthetic prompts and replies are counted as whole items. For each chunk or item, we divide its token count by o200k’s count on the exact same text. Within each text type, we take the arithmetic mean of these ratios, mean(t / t_ref). We then average text types equally within each language, and languages equally for the EU and overall results. Missing text types are omitted. Code is averaged separately.

We use o200k as the default reference because it is widely used and performs well across our test set. We also report average tokens per 1,000 characters. Per-type confidence intervals come from bootstrapping chunks or items; aggregate intervals approximate uncertainty in the mean by combining sampling error with variation between text types and, where applicable, languages.

Token counts for the non-public tokenizers are obtained through the provider’s API. Anthropic and Google both offer endpoints that count tokens without running the model, xAI’s Grok API offers a tokenize_text function that returns the tokenized sequence itself. We subtract estimated chat-message overhead to compare text-only counts. We confirm that the APIs agree with local tokenization in two checks: Gemini 3.x and the public Gemma 4 tokenizer return identical token counts, the same holds for GPT-5.6, GPT-6 Astra and gpt-oss.

3. Results

3.1 The Winners

Command A+ takes first place, leading in eight of the twelve languages, with Muse and Grok 4.6 coming in second and third overall. Initially, we expected which tokenizer needs the fewest tokens to depend on the language, but mostly it does not. Even in the three languages where Command A+ is not on the podium, it still ranks in the top-6. The list of great general tokenizers is completed by Muse, o200k and Gemini 3/Gemma 4, which always place in the top-10.

Chinese obviously goes to the Chinese labs, filling the entire podium, with Kimi being best followed by DeepSeek-V4 and Qwen3.8. European tokenizers shine on Polish. No tokenizer beats OpenAI’s o200k on Dutch; the runner-up, Muse, needs 6.7% more tokens.

3.2 Heavy Taxes

Claude 4.7+ needs more tokens than any other tokenizer we test in eight of the twelve languages, and never ranks better than 16th place. Compared to OpenAI’s models on German it uses 2x the amount of tokens! So, when comparing prices for the same German text, multiply Claude’s per-token rate by about two to put it on an o200k-equivalent basis. Over all languages Claude 4.7+ uses 65.3% more tokens, quite a hefty tax.

With Qwen being an exception, all Chinese model tokenizers are specialists. They are great for Chinese and match the reference on English, but on average use at least 10% more tokens for the EU languages we tested.

For German text, Kimi K3 uses 29.4% and Laguna S 2.1 42.2% more tokens compared to OpenAI models and Inkling from Thinking Machines.

3.3 Comparison Across Languages and Text Types

Below we visualize how tokenizer efficiency changes from one language to another, and from one kind of text to another.

Going from English to German, sixteen of the twenty tokenizers change rank. Kimi K3 is a top-five choice on English but only 17th on German. OpenEuroLLM 256k is below average for English, but moves from 16th to 2nd in German. Only Command A+, Muse and Grok 4.6 hold the top four at both ends.

How much do these averages depend on what you tokenize? Below you can select a tokenizer and language to see which text types have more or less overhead than the average marked by the dashed line.

Across all tokenizer-language pairs, text type usually has a modest effect, but there are substantial exceptions. For each pair, we compare the highest text-type ratio with the lowest (max / min - 1). The average spread is 12%, but the largest is 60%: Gemini 3.x/Gemma 4 on Japanese ranges from 0.728 on scientific text to 1.167 on political text.

3.4 Code

Compared to English text, 14 of the 20 tokenizers have a higher token-count ratio to o200k on source code. Claude 4.7+ needs 70.3% more tokens than the o200k reference for the same code. Command A+ is again the most efficient with 2.2% fewer tokens. Llama 3.x, Muse, GLM-5.3 and cl100k also beat o200k, by slight margins. The European-language-centric vocabularies tank a little bit on code. EuroLLM needs 43.5% more tokens for code, the two OpenEuroLLM tokenizers need 26.9% and 30.9% more tokens. Laguna S 2.1 is built for coding, but its tokenizer efficiency does not reflect that in our tests. It uses 11.1% more tokens for code than the reference, putting it in 14th place. Measured in tokens per 1,000 characters, SQL is the most expensive of the five programming languages for 16 of the 20 tokenizers.

4. Inside the Vocabularies

We analyze the published vocabulary files of tokenizers and attribute entries to a language or script.

We sort the entries in the 17 public vocabularies into seven buckets:

This lets us provide a rough description of what a vocabulary can represent. Note that this does not necessarily reflect what its training data contained, but it could be a hint.

Here are the vocabulary shares attributed to different languages and groups.

East Asian scripts include Han characters, Japanese kana and Korean hangul. We report them by script, because Han characters are shared between Chinese and Japanese and cannot be assigned to one of them.

The share clearly allocated to the seven European languages ranges from 1.7% (Kimi K3) to 15.6% (EuroLLM). Model developer origin predicts the shares quite reliably. The four Chinese tokenizers commit between 19.3% and 42.9% to the East Asian scripts, the two OpenEuroLLM vocabularies only commit 0.2% and 0.1% to them. Therefore, EuroLLM and OpenEuroLLM tokenizers allocated a relatively large number of entries to the 7 EU languages we tested.

Spending more entries on a language group does not by itself win it. Averaged across the seven EU languages we test, none of the four European tokenizers beats o200k. OpenEuroLLM 256k needs only 1.3% more tokens but has a 30% larger vocabulary; EuroLLM needs 5.6% more, Mistral’s Tekken 7.8% and OpenEuroLLM 128k 9.2%. EuroLLM beats o200k only on Polish. Both OpenEuroLLM vocabularies beat it on Polish and Italian, but lose on French and Dutch. Tekken loses all seven, French included. Note that OpenEuroLLM optimizes for 36 European languages, so the seven we test are a fraction of what its vocabulary has to cover.

The two vocabularies with the largest structural share, cl100k and Laguna S 2.1, each spend close to 40% of their entries there: about a quarter of the vocabulary on punctuation, digits and symbols, and the rest on short fragments and control tokens. Poolside builds its models primarily for coding applications, so this makes sense on first glance. However, in Section 3.4 we measured this directly and found the opposite. Laguna needs 11.1% more tokens for code than the reference o200k tokenizer and ranks only 14th of 20. So, a large structural share does not by itself make a vocabulary efficient on code.

The difference in vocabulary allocation explains most of what we measured in Section 3. Kimi K3 holds 570 German entries, 0.35% of its vocabulary, and under 30% of the one-thousand most frequent German words are a single token, which explains Kimi’s rather poor efficiency for German text.

Allocation priorities can outweigh vocab size. For example, DeepSeek-V4 has half the vocab size of Gemini 3.x/Gemma 4, 129,280 vs. 262,144, and still needs 15% fewer tokens on Chinese. DeepSeek spends 35,285 vocabulary entries on Han characters compared to 20,163 in the Gemini 3.x/Gemma 4 tokenizer.

5. Tokenizer Timeline

Tokenizers change slowly and are often re-used across model generations or versions. For example, Kimi K3 was released a year after Kimi K2 and uses the same vocabulary. Mistral’s Tekken v1 got introduced with their Nemo model mid-2024 and is still in use today, not only by Mistral themselves, but also in NVIDIA’s Nemotron 3 and 3.5, Apertus, and SOOFI. Even OpenAI does seemingly not have any reason to upgrade their o200k tokenizer, which still powers their released frontier models.

Vocabulary sizes do not always grow over time. Reconstructions suggest that Anthropic shrank Claude’s vocabulary, from 65,000 entries for Claude 1 and 2 to an estimated 49,152 for Claude 3 through 4.6 and 16,384 with Claude 4.7. These later sizes are inferred rather than published by Anthropic; see Land’s reconstruction. The smallest estimate is roughly a twelfth of OpenAI’s o200k vocabulary size. Z.ai went the other way after an early cut, from a 130,344 token vocab in ChatGLM to 64,789 in ChatGLM2 and 3, then to 151,343 from GLM-4 onwards. Google and Cohere kept their tokenizer sizes pretty much constant.

6. Limitations and Caveats

Tokenizer efficiency is not model quality. How many tokens a text is split into does not necessarily tell you much about how well a model handles it. How well a model reasons, writes or follows instructions in a language may not correlate with tokenizer efficiency at all. A model with a less efficient tokenizer can still be the better choice for a task. Anthropic manages to push the performance frontier despite poor tokenizer efficiency. That makes its apparent move toward a much smaller vocabulary especially interesting. Could smaller vocabularies improve optimization stability, encourage more sharing across languages, or offer another benefit that outweighs the extra tokens? Our measurements cannot answer that, but we would love to know what readers think Anthropic gains from this choice.

A vocabulary entry does not guarantee the model uses it well. In Section 4 we count what different vocabularies contain. However, a vocabulary fitted to one dataset and then used to train a model on another dataset can result in entries the model rarely sees. Land and Bartolo (2024) call these under-trained tokens and show they are widespread, the canonical example being SolidGoldMagikarp in GPT-2 and GPT-3. This is a known risk of inheriting tokenizer vocabularies.

Our test datasets are not parallel translations. Each language ratio is measured against the reference tokenizer. This avoids translationese, at the cost of weakening direct cross-language claims. That is why Section 3.3 compares rankings and reference-relative ratios within each language, rather than treating texts in different languages as equivalent.

Our ratios measure compression, not prediction. For each tokenizer we report token count divided by the reference tokenizer’s token count on the same text. This is also called normalized sequence length in the literature. Bits per byte is a metric for comparing LLMs that do not share a vocabulary. It multiplies a model’s loss per token in bits by the tokenizer’s tokens per byte and here we measure only the second of those. Our token-count ratio is independent of characters and bytes: neither appears in t / t_ref. The absolute density axis in the interactive figure does depend on that choice. In a separate check of the locally measured prose corpora, ranking tokenizers by average tokens per byte instead of average tokens per character swapped two adjacent pairs: one in Arabic and one in Japanese. That is a comparison of absolute densities; the reference-relative ratios do not change. For example, a Japanese character uses about 2.7 UTF-8 bytes. In these prose corpora, Japanese costs 3.48x English per character on the reference tokenizer, and 1.33x per byte. In the end we chose to report characters because it is the unit someone types and can count. Across scripts neither unit settles the question, since a hundred Japanese characters and a hundred German ones are not the same message, and that comparison needs parallel text for the reasons above.

Part of our test dataset is model-generated. Our synthetic items include prompts and replies generated around real snippets, since we wanted to test assistant-style text alongside plain documents.

7. Takeaways

• Command A+ tests best overall. It wins eight of the twelve languages. Muse and Grok 4.6 are the next best general tokenizers in our test.
• Claude 4.7+ uses the most tokens, by far. 2.01x the reference on German and 65.3% more across all twelve languages. It ranks last in eight of the twelve languages we test.
• Tokenizer choice can make roughly a 2x difference. On the same German text, the least efficient tokenizer produces 2.1x as many tokens as the most efficient.
• Text type usually has a modest effect, with exceptions. The spread between the highest and lowest text-type ratios averages 12% within a language, but reaches 60% in the largest case.
• Specialists are excellent in few languages and among the worst in others. Kimi K3 is first on Chinese and seventeenth on German.
• English-only workloads are not exempt from the tokenization tax. Compared to OpenAI’s models, Claude 4.7+ uses 58.6% more tokens on English.
• Why create your own tokenizer? OpenAI reached GPT-6 Astra level capability with a tokenizer they shipped in May 2024. Thinking Machines also uses o200k for their Inkling models and decided against building their own tokenizer. None of the four European tokenizers beat o200k when averaged across the seven EU languages we tested, and they are also worse than o200k on code! Better support for languages poorly covered by existing tokenizers can be a good reason to roll with a custom tokenizer, but you should be aware of the tradeoffs and carefully consider whether it is worth it. Getting rid of potentially under-trained tokens can be another reason to go with a custom vocabulary, but why not just prune an existing vocabulary?

Did We Forget Your Favorite Tokenizer?

If a tokenizer you care about is missing, or you want results for a language we did not cover here, reach out.

Acknowledgements

• This work is supported by the OpenEuroLLM project, co-funded by the Digital Europe Programme under GA no. 101195233.
• This work is supported by the LLMs4EU project, co-funded by the Digital Europe Programme under GA no. 101198470.
• This work is supported by the German Federal Ministry for Economic Affairs and Energy (BMWE) through EU-SAI/SOOFI: Sovereign Open Source Foundation Models for European Intelligence (grant number 13IPC040J).
• This work is supported by the State of North Rhine-Westphalia and the European Union under the NEXT.IN.NRW innovation program, through the LLM4KMU project.