Kimi K2, measured: the 1 trillion-parameter model everyone is benchmarking is mid-pack on the language tax.
Tetracta AI Teams · 21 July 2026 · a companion to the 204-language token-tax leaderboard
Kimi K2 is the model of the week, and K3 is already being hyped as an Opus-class open release. So we did what we always do with a model everyone is talking about: we measured the one thing you can verify yourself. Not a benchmark score you have to trust — the tokenizer. Kimi K2's tokenizer is public, so we ran it through the exact same 204-language corpus as our existing leaderboard and asked a narrow, answerable question: how many tokens does Kimi charge for the same sentence, in every language?
The answer is not a scandal and it is not a coronation. It is a number, and the number is ordinary.
Where it lands: 4th of 9
Average tokens-per-sentence relative to English, across 204 languages on FLORES-200. Lower is better. Kimi K2 slots in behind Gemma-2, BLOOM and DeepSeek-V3 — and, notably, the models that win multilingual efficiency are not the frontier names people benchmark.
The one genuinely interesting thing: a bigger vocabulary that didn't pay off
Kimi K2 is built on the DeepSeek-V3 architecture — the config literally instantiates it. But Moonshot gave it its own tokenizer with a 28% larger vocabulary (163k vs 128k). A larger vocabulary is the standard lever for cutting the multilingual token tax: more subwords means fewer pieces per word. So the reasonable expectation is that Kimi should be more token-efficient than DeepSeek-V3.
It is not. Net across 204 languages, Kimi K2 lands a hair behind its own ancestor (2.52× vs 2.49×), winning as the cheaper of the two in only 63 of 203 languages. The extra vocabulary went somewhere — mostly into Chinese, where Kimi is the single most efficient tokenizer we measured (0.89×, cheaper than English) — but it did not translate into a broad multilingual win.
Head-to-head with the model it's built on
Token tax vs English (same FLORES-200 sentences). Kimi K2 vs its DeepSeek-V3 ancestor, with Gemma-2 (the leaderboard leader) for reference. The last column marks the cheaper of Kimi and DeepSeek, and by how much. The pattern is a trade, not a rout: Kimi is dramatically worse on some scripts (Thai +98%, Bengali +48%) and clearly better on others (Burmese −34%, Amharic −29%).
| Kimi K2 | DeepSeek-V3 | Gemma-2 | cheaper of the two | |
|---|---|---|---|---|
| Chinese | 0.89 | 0.96 | 1.09 | Kimi −7% |
| German | 1.72 | 1.51 | 1.25 | DeepSeek +14% |
| French | 1.72 | 1.53 | 1.34 | DeepSeek +12% |
| Spanish | 1.70 | 1.50 | 1.26 | DeepSeek +13% |
| Russian | 2.08 | 1.61 | 1.40 | DeepSeek +29% |
| Arabic | 2.06 | 1.62 | 1.50 | DeepSeek +27% |
| Turkish | 1.94 | 2.02 | 1.39 | Kimi −4% |
| Hindi | 2.76 | 2.82 | 1.86 | Kimi −2% |
| Japanese | 1.65 | 1.43 | 1.17 | DeepSeek +15% |
| Korean | 1.98 | 1.66 | 1.64 | DeepSeek +19% |
| Vietnamese | 2.79 | 2.11 | 1.36 | DeepSeek +32% |
| Thai | 3.53 | 1.78 | 1.83 | DeepSeek +98% |
| Bengali | 2.98 | 2.02 | 2.69 | DeepSeek +48% |
| Burmese | 3.51 | 5.30 | 4.79 | Kimi −34% |
| Amharic | 4.31 | 6.07 | 3.19 | Kimi −29% |
What this means for your bill — and what it does not mean
If you build in Chinese, Kimi K2's tokenizer is genuinely excellent and will stretch your context window and cut your per-token cost. If you build in Thai, Vietnamese, Bengali, Russian or Arabic, DeepSeek-V3's tokenizer is the cheaper of the two — and Gemma-2's is cheaper than both. That is a real, spendable difference: token count is the billing unit, the GPU-time unit, and the true size of your context window.
Now the honest fences, because this is the part the hype skips in both directions:
- This is efficiency, not quality. A token tax says nothing about whether Kimi K2 is good at coding, agents or reasoning — which is what it is actually being hyped for. A model can be token-expensive in a language and still be excellent at it. We measured cost, not capability, and we do not conflate the two.
- Mid-pack is not a failure. 4th of 9 on a hard multilingual metric is a perfectly respectable tokenizer. We are not calling Kimi bad; we are declining to call an ordinary result extraordinary. The interesting part is narrow and specific: the bigger vocabulary did not beat its own ancestor.
- We did not open the model up. Our deeper X-Ray reads a model's internal weights, and a 1-trillion-parameter mixture-of-experts does not fit the single-GPU pods we scan on — and K3's weights are not public yet. So this is the tokenizer layer only. When K3 ships, we will measure its tokenizer the same way.
- One corpus. FLORES-200 is news and encyclopedic prose. Code, chat and domain text tokenize differently; a Chinese-first model like Kimi may look different again on informal or technical text. The method is public so you can re-run it on your own.
Method (reproducible)
Corpus: FLORES-200 dev — 997 sentences professionally translated into ~200 languages,
so each language measures the same meaning. Metric: total tokens for a language ÷ total tokens for the
identical English sentences, per tokenizer (add_special_tokens=False). Tokenizers: Kimi K2
(moonshotai/Kimi-K2-Instruct) plus the eight already on our leaderboard (Qwen2.5, Llama-3.1, Llama-2,
Mistral-v0.3, Gemma-2, DeepSeek-V3, BLOOM, Phi-3.5), all public and unmodified. Token count is a property of the
tokenizer, so it is identical across model sizes within a family and needs no weights and no GPU — which is why
this is a $0 measurement anyone can reproduce. The internal X-Ray scan is a separate, deeper instrument.
See the full 204-language leaderboard → X-Ray for LLMs
— Tetracta AI Teams · for humans, like humans. We measure; we don't hype.