Research and evidence

Status at a glance

Every item on this page carries one status tag. The tag tells you how far to trust the result.

• Current The result stands as published, within the scope stated on its page.
• Provisional Measured, and pre-registered or reproducible, but at diagnostic scale or with a small sample. We expect to test it again at larger scale.
• Withdrawn No longer evidence. The page stays online, marked as withdrawn, so that old links and citations still resolve.
• Open beta A product in public testing. Its outputs are descriptive and have not been validated.

Rational Attention Provisional

Rational Attention is our drop-in replacement for softmax attention. It adds zero parameters. We do not disclose how it works; we publish what it does when we measure it, and the data files to check our numbers.

The comparison holds everything else fixed: the same model body, optimizer, seed and data, with no stabilisers on either side, at a high learning rate. We ran it at about 601M, 1B, 3B and 7B parameters on H100 and H200 GPUs. The 3B and 7B runs were short diagnostic runs of at most 236M tokens.

Training stability against softmax

Cpk is the process-capability index from manufacturing quality control: it says how consistently a process stays inside its limits. By common convention, 1.33 or more is called capable and below 1 is out of specification. Our note on gradient-norm control charts explains the method; the control-limit setting is not disclosed.

Rational Attention costs about the same as softmax and stays stable where bare softmax does not. It is no faster. The claim holds only at the diagnostic scale above.

Loss: a small edge from learning-rate headroom

At 3B and 7B, our best Rational Attention run reached a slightly lower held-out loss than our best softmax run. We read this as a direction only, and we attribute it to learning-rate headroom: Rational Attention trains at learning rates where bare softmax fails. When we ran both at the same learning rate with both stabilised, softmax came out 0.0153 bpb ahead, the same size as the edge. We call that parity. With two sizes we cannot fit a scaling law.

Effect on hallucination: tested, null

We tested whether Rational Attention reduces hallucination. A controlled test came back null, so we claim no such effect. The test has not been run at behavioural scale.

Scale, and what is not done yet

• Every result above is diagnostic scale (sub-Chinchilla) and provisional. None of it is a production training result.
• Not run yet: a properly budgeted (Chinchilla-sufficient) 3B validation with pre-registered pass and stop thresholds; downstream task quality; the hallucination test at behavioural scale.
• Compatibility was tested with two optimisers (Muon and AdamW), mixture-of-experts models, bf16 and multi-GPU training. The full list of configurations is in the technical brief. That does not establish compatibility with every architecture, kernel or training pipeline.
• Our quantiser for this operator is in development. On a 3B diagnostic checkpoint, int8 changed held-out loss by 0.0000 bpb and int4 by +0.0005 bpb. The saving is in memory only.
• Other teams have published alternatives to softmax attention; we discuss related work in the technical brief under NDA.

Evidence files

The public bundle holds the full data: every experiment, every training step we evaluated and every stability statistic. What we withhold is the recipe: the formula, kernel internals, exact learning rates, schedule and SPC thresholds.

Z-Next Current

Z-Next is our own language-model architecture. It has no attention layer and no KV cache. Everything it holds about a conversation lives in a fixed-size state of about 5.2 MiB, and the size is enforced in code. When the input grew from 952 to 31,926 tokens, the state stayed byte-identical.

The v4.3 base model has about 947 million parameters. We trained it from scratch on 10.0B tokens of a FineWeb-Edu sample and English Wikipedia with a 64k SentencePiece tokenizer, in one continuous run of 29.97 hours on 4 H200 GPUs, with no divergence and no intervention. It is English only. Validation perplexity is 15.1 at a context of 512 tokens.

The twin shares the corpus and tokenizer but not the exact training recipe: a small share of Z-Next's pre-training windows also carried program-generated material that the transformer did not see. The base result therefore shows that removing attention cost no measurable quality. It does not show an architectural advantage. After instruction tuning, most of the gap comes from the twin dropping (LAMBADA −8.50, BoolQ −5.63 points), while Z-Next itself gained about 0.5.

The product variant scores 45.34. It passed 14 of 17 pre-registered gates, so it is recorded as a fail. It is still our best chat variant, so it runs the demos, labelled as not passing.

Not solved: long-context recall is near chance (0.306 against 0.25) at 1B parameters and 10B training tokens, as it is for the same-size transformer. Not done: anything beyond 1B, a matched comparison with other fixed-state models, independent reproduction, safety tuning. The model card has the full list. Weights are not public.

• Z-Next 1B model card: details, evaluation, limitations and access.
• Z-Next model card: measured 17–21 Aug 2026, with a follow-up on 23 Aug 2026. It replaces our older technical page and corrects one statement in it (below).
• Z-Next system card and VAN-1B model card, and every other model on the models index.

Studies and notes

Every research note we have published, newest first. Each keeps its original address.

The Z-Next technical report is summarised above, and the Rational Attention data is in its evidence files. Notes we withdrew are listed in the next section.

Withdrawn and corrected

When a result does not survive our checks, we say what failed, mark the page and keep its address online so citations do not break. Nothing in this section is current evidence.

Model X-Ray legacy findings Withdrawn

The interpretations we once published from Model X-Ray outputs were withdrawn on 4 September 2026, with follow-up corrections to 6 September. The correction record keeps the full dated record.

Model X-Ray runs today as an Open beta for eligible public models up to the 7B class, with payments off. Its reports are descriptive and are not validated diagnostics. The Model X-Ray status page (last entry 11 Nov 2026) describes the beta's current scope.

Other retired or corrected claims

• Chimera-1 and Model X-Ray. The merge note once said that Model X-Ray measurements predicted merge success and guided which skills were kept. That interpretation is withdrawn. The merge measurements, the baseline comparison, the general checks and the negative control are retained. The note's rule of thumb for when merges help is a working hypothesis from one campaign.
• Frankenstein merge note. Withdrawn Our earlier merge write-up at /note-frankenstein-merge.html is retired. Its address now forwards to the Chimera-1 note, which replaced it.
• Language notes. The token tax, Kimi K2 and language-generation notes had Model X-Ray references and sales wording removed in the September correction. Their tokenizer and length results did not depend on Model X-Ray and are retained.
• Rational Attention loss edge. We no longer describe the small loss edge as an architectural advantage. With both sides stabilised at the same learning rate it is parity, as shown above. Our control-chart note describes an earlier headline edge of ours that turned out to be a baseline collapsing at a learning rate it could not tolerate.
• Z-Next public statements. 10 Nov 2026: the Hugging Face card was corrected on three points to match the model card on this site: the description of the training data, the benchmark status of the chat variant, and the removal of the session-count comparison. Our older technical page said no chat-tuned build had run the classic benchmarks; the product variant has, and the model card, which replaces that page, gives its score. The report's earlier Model X-Ray interpretation was removed in September.
• Sessions per GPU. An earlier home page compared how many idle sessions fit on one 16 GB GPU for Z-Next and its twin. It covered one configuration, against a twin whose cache was not optimised, so we no longer show it.

How we run studies

The mechanisms stay private; the way we test is below. The methodology one-pager has the short version.

Pre-registration

Before a run, we write down the question, the metric, the comparison and the rule for pass or fail. The rule does not change after the run: under such a rule the Z-Next product variant is recorded as a fail.

Matched comparisons

The two sides of a comparison share the model body, optimizer, seed and data, and we compare them at the same training step. An unmatched comparison once showed us a gap of about 0.06 bpb; at the same step the real gap was about 0.002 bpb.

Deterministic recompute and sealed result files

When a run finishes, we seal its result files by recording their md5 checksums. Our headline numbers regenerate deterministically from those sealed files; we do not copy them by hand from notes or slides. Anyone can check the public files against the manifest.

Seed-null and negative controls

Before we call a difference an effect, we compare it with the difference we get by changing only the random seed, or with the run-to-run noise of the setup. In the Chimera-1 study that noise was about ±2 points, so we do not interpret differences of that size. We also keep negative controls: in Chimera-1, a merge with a parent that had drifted far from the shared base, which collapsed to unusable output; and in the kernel check, a deliberately corrupted variant that the check correctly rejects.

Stability as a number

We log the gradient norm on a control chart and report stability as a capability index and an excursion count.

Nulls and failures published so far

The hallucination test that came back null, the product variant that failed its own gates, three merge variants that did not beat the plain baseline, a data-pipeline artifact that faked a 0.11 bpb edge, and the Model X-Ray withdrawal above.