Language Economy · pre-registered · reproducible

Two layers of the language tax: the tokenizer, and how much the model says.

Tetracta AI Teams · 21 July 2026 · a follow-up to the 204-language tokenizer leaderboard

Our token-economy leaderboard measured one thing: how many tokens a model's tokenizer turns the same text into, per language. That is the input tax — real, but it is only half the bill. The other half is what the model generates: how many tokens it actually spends to answer in each language. So we measured that too — and pre-registered the question before looking, because a finding you want to be true is exactly the kind you should try hardest to break.

Two effects fall out, and they are cleanly separable. One is boring and rock-solid; the other is more interesting and more honestly hedged.

Layer 1 — the tokenizer tax does not change with model size

Tokens-per-output-character is a property of the tokenizer, not the trained weights. So it should be identical whether the model is 0.5B or 32B — same family, same tokenizer. It is:

Line chart of tokens per output character across Qwen2.5 sizes 0.5B to 32B: every language forms a flat horizontal line — Hindi near 0.90, Chinese near 0.60, English near 0.20 — unchanged by model size
Tokens per output character, Qwen2.5 0.5B→32B. Flat lines: the per-language token density of the output is a tokenizer constant (Hindi ≈0.90, Chinese ≈0.60, English ≈0.20), unmoved by scale — the same size-independence we found on the input side, confirmed from a completely different angle.

Layer 2 — every model answers in non-English at a fraction of its English length

The generation side is where behavior, not arithmetic, takes over. We asked each model the same 12 questions, once in English and once instructed to answer in each of six other languages, and measured the length of the answer (in characters, so the tokenizer tax doesn't contaminate it). The result is consistent and it holds across two independent families:

Two panels (Qwen and Llama) of answer length relative to English across model sizes: for both families and all sizes, non-English answers sit well below 1.0 — roughly 0.3 to 0.7 of the English length — with Chinese lowest
Answer length in each language relative to the same model's English answer (1.0 = same length). Across Qwen (0.5B–32B) and Llama (1B–8B), non-English answers land at roughly 0.3–0.7× the English length — the model is systematically terser in other languages. Chinese is the extreme (~0.15–0.25×), reflecting a genuinely denser script.

The headline that survives is simple: models are more verbose in English than in any other language we tested, in both families, at every size. Combined with Layer 1, the practical effect on your bill is the product of the two — a per-language answer costs (its length) × (its token density).

What we do not claim — the honest limits

  • This is not a clean scaling law. We would have liked to write "bigger models get more English-centric," and there is a mild pull in that direction — but the smallest models (0.5B, 1B) are erratic (they over-produce in some languages), and the trend across capable models (3B and up) is gentle and noisy, not monotonic. We are not going to dress up a 2–7-point, single-run curve as a law.
  • Shorter is not the same as worse. We filtered out answers that were not actually in the target language, so the models are answering in-language — but "terser" could mean "more concise" or "less thorough," and this measurement cannot tell them apart. It is a length result, not a quality verdict.
  • Small n, two families. Twelve questions, greedy decoding, Qwen2.5 and Llama-3. Hindi in particular is under-sampled on the smaller models (its answers are so token-heavy they hit our generation budget), so read its smaller-model points with caution. A broader study — more families, more prompts, a quality control — is a separate, larger piece of work, and we say so rather than imply we have done it.
  • Efficiency, not quality — and a companion to the internal scan, not the scan itself. As with the leaderboard, this is about cost and effective context, and it reads only public tokenizers and outputs, not the model's internals.

Method (reproducible)

Models: Qwen2.5-Instruct 0.5B / 3B / 7B / 32B and Llama-3 Instruct 1B / 3B / 8B, fp16, greedy decoding (temperature 0), 1024-token budget. Prompts: 12 fixed general-knowledge questions, asked in English and, for other languages, prefixed with "Answer in <language>." Per (model, language) we record output tokens and output characters, keeping only generations that finished naturally (did not hit the token budget) and passed a script-based language-adherence check (non-Latin scripts must be present; Latin-script outputs must be non-degenerate). Metrics: answer length vs the same model's English answer (characters) and tokens-per-character. The 32B point ran on a single H200 cloud GPU, provisioned per job and destroyed after, with the balance verified afterward; all other sizes ran locally. Pre-registration ID: PR-OUT1. This is a length/economy measurement; the internal X-Ray scan is a separate, deeper instrument.

See the 204-language tokenizer leaderboard → X-Ray for LLMs

— Tetracta AI Teams · for humans, like humans.