Twin Loss Curves, One Operator Apart: A Matched-Pair Study with Seed-Null Controls
Tetracta AI Teams · published 10 July 2026 · substantially corrected 13 July 2026
Here is the uncomfortable question we started with. Two models sit in front of you. Same data, same order, same seed, same number of steps. Their validation loss agrees to within a fifth of a percent. Their sample outputs match. Every instrument you own says: these are the same model. How would you know if they were not — and how would you know whether any difference you found was real, or just the luck of the draw between two training runs?
What we built, and what we first claimed.
We trained two decoder-only transformers at 0.93B parameters — twenty-four layers, grouped-query attention, RoPE, SwiGLU, a sparse mixture-of-experts, a tied head. Identical initialization seed, identical tokenized corpus streamed in identical order, identical optimizer, schedule and step count. One component differs, and only one: the function that turns attention scores into weights. One arm uses standard softmax ("VAN"); the other uses Tetracta's rational attention ("RAT"), a softmax-free operator that adds zero parameters and whose form we do not disclose.
At step 30,000 of a 157,000-step plan the two arms sat at validation bits-per-byte of 1.0092 (VAN) and 1.0102 (RAT) — a fifth-of-a-percent tie. We then scanned both with our own instrument, Model X-Ray, which injects a calibrated disturbance inside the network and watches it propagate, layer by layer, without ever grading the output. Inside, the two looked different — and we published four differential findings, plus one null.
The reviewer's objection: where is the denominator?
An external critic reduced it to a single question. Every one of those magnitude claims needs a control: how much do two runs of the same recipe, differing only in their random seed, already diverge inside? Our metric was relative, and it came from a single matched pair. If two ordinary softmax runs already diverge that much internally, then what we had measured was not "the operator" — it was the noise floor between any two runs. The right response is not to argue. It is to measure. And the thresholds are sealed before the run, so the answer cannot be talked into being favorable.
The control: three new models, one sealed question.
We trained three more models with the identical recipe — only the seed changed: VAN-seed-B and VAN-seed-C (two more softmax runs, to establish the noise floor) and RAT-seed-B (to see if the rational-arm result replicates). Five checkpoints in all, each scanned by identical code, each number re-derived from raw outputs. Roughly $300 of compute; every pod terminated afterward; every checkpoint md5-sealed.
What the control showed.
The loss twins are real — and the control made that claim stronger. The three softmax runs landed at bpb 1.0092, 1.0072 and 1.0085; the two rational runs at 1.0102 and 1.0101. The spread between softmax seeds (0.0020 bpb) is larger than the original softmax-vs-rational gap (0.0010). The operator moves the loss by less than the seed does. A leaderboard genuinely cannot separate them — now with a control to prove it.
The "4.4× quieter inside" figure collapsed. Internal disturbance under int8 damage, per model: VAN-42 4.166, VAN-B 0.573, VAN-C 0.967, RAT-42 1.277, RAT-B 1.104. The spread among softmax seeds alone is 7.3×. The original VAN-42 was simply a high outlier; the softmax median (0.97) sits below the rational median (1.19). The direction of the effect does not even survive — let alone the magnitude.
The routing-stability edge was noise. int8 token-level routing flips: VAN-42 2.97%, VAN-B 0.93%, VAN-C 1.74%, RAT-42 0.52%, RAT-B 2.66%. The rational replication lands squarely inside the softmax band; RAT-42's low value was a lucky draw. Softmax seed spread: 3.2×.
The "84% of depth" divergence was a ceiling. Cross-scan mean deviation: the operator pair (VAN-42 vs RAT-42) reads 0.000307; the seed pair (VAN-B vs VAN-C) reads 0.000324 — both spanning 84% of depth, a ratio of 1.06×. Two ordinary softmax runs diverge internally exactly as much as the operator swap does. (Our own note had flagged 84% as a "ceiling indicator that saturates for any sufficiently different pair." The control confirmed it, at our expense.)
Finding four, unchanged: the null we kept.
The one finding that has not moved is the one we never got. We had wondered whether the rational arm abstains from fabrication more readily. It does not — at least not measurably here. At step 30k both models know essentially zero facts, so there is no knowledge signal for an abstention to act on; the apparent gap was an artifact of an 8-token generation budget. We published that null on day one, and it is still a null. A diagnostics lab that publishes only the findings flattering its own architecture has quietly stopped being a lab.
What still stands.
1 · The instrument works on a custom architecture. These are not HuggingFace models. A thin loader let our standard product pipeline scan a bespoke, undisclosed operator unchanged — the capability the retraction does not touch, and the one that matters for a diagnostics product.
2 · A separate, seed-independent finding held up. On a converged public mixture-of-experts model, real int8 kernels flip about twice as many routing decisions as the round-to-nearest simulation that most quantization evaluations use (~2.6% vs ~1.3%, deterministic over 1.47M decisions). Anyone auditing quantization with a simulation is seeing roughly half of the real kernel's effect on routing. That result does not depend on our twins, and it stands.
3 · The method is the point. Pre-registered thresholds, deterministic recompute, md5-sealed evidence, and a negative control that we let overturn our own headline. During this work a later analysis even produced a beautiful ρ=0.95 correlation that would have "saved" a product story — and we ran it down to a tautology and discarded it, because a correlation you don't decompose is a press release, not a finding.
Why we are telling you this.
The narrow lesson is that our own headline did not survive its own control. The broad one belongs to anyone shipping models: output-level equality is not evidence of internal equality — but neither is a single internal measurement evidence of a real effect. The gap between what a model says and what it is, is exactly where deployment risk lives; and the discipline that separates a real signal from the luck of a seed is exactly the service we sell. This note is that discipline, run on ourselves, in public. We would rather find a confound ourselves, with a pre-registered control, than have a reader find it for us.
Full data companion. All measurement tables from this campaign — seed floors, jitter floors, the real-vs-simulated int8 battery and the discarded ρ=0.95 — are published separately: Noise floors for internal LLM measurements.
Artifacts. The checkpoints, X-ray summaries, brain-portraits and the equal-damage weight-error script are published on Hugging Face as a provenance artifact — the numbers can be checked against sha256-sealed files, while the analysis instrument stays closed. The rational operator itself is not disclosed; a method patent application is in preparation. Loading the rational weights into a softmax model will produce meaningless outputs — please do not benchmark that and report it as ours.
— Tetracta AI Teams · for humans, like humans.
Download checkpoints & scans ↗← All research notes Talk methodology