You fine-tune, quantize, merge or distill a model you ship — and the eval score barely moves. That does not mean nothing moved. X-Ray reads the model layer by layer: where the change landed, and whether knowledge quietly regressed — facts that broke, and facts the model still answers correctly while its internal confidence has collapsed. Output-only evals miss that last one by construction.
Request a free trial seat → 20 seats · selections decided by our own model
A layer-by-layer map of the difference between two checkpoints — base vs fine-tune, fp16 vs quantized, pre vs post merge. The report names the contiguous band holding most of the change, so you take an address to your evals, not a guess.
The knowledge probe scores facts the model knows against fabrications it should refuse — inside the layers. It catches facts that broke, facts that recovered, and facts still answered correctly but with collapsed internal confidence.
Every scan ships an Ed25519-signed attestation: weight fingerprint, compute region, deletion record. Change one character and the signature fails. Verify it offline against our published key — you don't take our word for it.
Models are fetched, scanned and wiped on a single-purpose worker. safetensors only (pickle refused). Private jobs run on wipe-guaranteed cloud GPUs. No training on your weights, no telemetry.
Every report ships a Language Economy panel: how many tokens your model spends per language versus English (API cost + context) — from its tokenizer alone, no extra scan. Efficiency, not quality. See below →
Headroom, complementarity and proximity are measurable from the inside, before you merge. We proved it in public: Chimera-1, two skills stitched into one 3B model — implanted skills +8.8/+25.0, and 1,210 standard questions flat everywhere else. The surgical profile →
Across the Qwen2.5 family, from 0.5B to 32B, fine-tuning drives change into a narrow band of layers while quantization spreads it wide and shallow — two visibly different internal signatures, on two independent metrics, every run pre-registered. Then we spent three weeks trying to break it: an instrument-floor control killed our first onset claim, a normalization test challenged the band, and cross-family runs (Llama, Mistral, Gemma) scoped the headline honestly to where it clears its own measured noise band. What survived, survived a falsification attempt.
Two things we measured before making this a bigger claim. First, we tested it beyond Qwen: in normalized space the contrast clears its own measured noise band in a second family (Llama, two scales) too — while a third (Mistral) points the same way but stays inside the band, so we log it as inconclusive rather than over-claim. A fourth (Gemma-2-2B) missed its pre-registered conditions outright — so the claim stays two-family, and the miss is published with the rest. Second, we checked it against a real int8 kernel (bitsandbytes) at 7B, not just our simulation: the real kernel moves internals more, and the fine-tune-vs-quant contrast is sharper under it — the simulation understates the effect, if anything.
Read the full autopsy — six attempts, four families, misses included →
We build models too. We trained two 0.929B twins — identical seed, data and order, differing only in the attention nonlinearity (a standard softmax vs our rational-attention, RAT) — and put both through int8. The outputs stayed byte-identical: zero behavior change on either. Their internal response did not match. That gap is exactly the thing an output-only benchmark cannot see, and exactly what the X-ray is for.
The twin study — matched-pair, seed-null controlled → · All 8 checkpoints + seed controls on Hugging Face →
We ran the knowledge probe across two generations of the same small model — Qwen2-0.5B-Instruct to Qwen2.5-0.5B-Instruct, same family, same chat format. On the aggregate score it looks like a quiet win: 14/20 known facts correct became 15/20. Underneath, the upgrade traded knowledge — two facts it used to get right, it now gets wrong; three it newly recovered. One example a release note would never mention: it used to answer “the capital of Italy is Rome,” and after the upgrade it answers “Florence.”
This is the report a team gets when they re-scan the same model before and after a training run — the second scan reads from the first, so it costs nothing.
The same sentence becomes a different number of tokens in every model — and tokens are what you pay for: API price, GPU time, and the real size of your context window. Every X-Ray report now includes a Language Economy panel for the model you scanned: how many tokens it spends per language versus English. It needs no weights and no GPU — just the model's own tokenizer — so it works for any model, including a custom one.
Off English the tax runs 2–3× typically, past 5× for some scripts, and up to 10× at the extreme. A model with a 1.55× German tax means German users pay ~55% more per API call, self-hosting burns ~55% more GPU per request, and a “128k context window” holds ~55% less German. This is exactly the kind of hidden cost a team building or shipping a multilingual model needs to see — before the invoice does.
What we are and aren’t claiming: this is an efficiency measurement, not a quality one — a high tax means a model’s tokenizer is inefficient for that language (more cost, less context), not that the model is “bad” at it. And it’s a tokenizer measurement, the lightweight companion to the internal X-Ray scan — not the internal scan itself.
Model merging is everywhere — and almost always done blind: merge, eval, hope. We used internal measurement the other way around: headroom, complementarity and proximity are all measurable from the weights and activations before you merge, and across ~10 experiments they predicted the outcome every time — including a negative control where an aggressively merged coder collapses to 0% on code.
The proof-of-concept is public: Chimera-1, two specialists stitched into one 3B model with a training-free merge. The two implanted skills jumped; a 1,210-question head-to-head against the base — MMLU, ARC-Challenge, HellaSwag, Belebele in five languages — came back flat everywhere else:
Honesty note: three attempts at internally-guided merge surgery did not beat the standard blind recipe on average — we publish that too. What internal measurement reliably delivers today is the diagnosis: which pair will merge profitably, before you spend anything on it.
Teams that fine-tune on a cadence, quantize for deployment, merge checkpoints, or distill for edge — and need to know what moved before it reaches production. Bring the same model before and after your next training run; the comparison reads the delta for you, and it costs nothing to re-scan against a stored report.
We're opening 20 free trial seats. Who gets in is decided by our own model — the same rational-attention architecture the X-ray was built to read. Applications reviewed weekly; we reply by email.