Z-Next: a language model with no attention, no KV cache, and a session state that never grows
A 947,153,897-parameter decoder-only language model, pre-trained from scratch on 10.00 billion tokens in 29.97 hours on 4×H200. It carries a conversation in a fixed-size state of 5.227 MiB that is the same size at token 1 and at token 32,000. On ten standard zero-shot benchmarks it scores 45.2 macro as a base model — ahead of budget-matched Pythia-1B (42.4) and Pythia-1.4B (43.5) by margins whose confidence intervals clear zero, and level with a same-data transformer twin (44.6): that last gap is +0.62 with a 95% interval of [−0.06, +1.28], which contains zero, so we call it a tie rather than a win. All of it while having seen fewer tokens than any of them.
What it is, what class it belongs to, and what it is for
What it is. Z-Next is a from-scratch autoregressive language-model architecture. It reads a prompt and writes a continuation, exactly like a transformer decoder does, and it uses the same tokenizer and the same training corpus as the transformer we compare it against. What it does not have is attention, and therefore it does not have a KV cache. Everything the model knows about the conversation so far lives in a state of fixed size.
What class it belongs to. On the standard map of sequence models, Z-Next sits on the constant-state side — the side occupied by recurrent networks and by the state-space family — and not on the attention side. It shares that family's defining property: inference cost per token does not depend on how much text came before, and memory per session does not grow with the conversation. It differs from the published members of that family in how the state is maintained. That part is our own work and we do not describe it here. Nothing on this page depends on you taking our word for the mechanism; every claim below is a measurement you could design yourself, and several of them you can reproduce on the public demo.
“So it’s Mamba with a new name.” No — and also, we are not claiming to have invented constant-state sequence modelling. That family exists, it is published, and Mamba and RWKV are its best-known members. We are claiming something narrower and checkable: a specific set of measurements that, as far as we can tell, are not published elsewhere in this form. Four of them:
- A same-data transformer twin, trained by us, as a control. Architecture papers are usually compared against public checkpoints trained by other teams on other corpora with other tokenizers and other recipes. We trained a deliberately ordinary transformer on the same corpus, same tokenizer, same budget, same harness, and we publish it — including the rows where it beats us. That control is what makes the comparison mean anything.
- Robustness to instruction tuning appears to differ by architecture. One recipe, applied unchanged to three architectures, moved them in three directions: our model +0.51 pp (95% CI [+0.04, +0.98]), our previous generation −0.43 ([−0.89, +0.04]), the transformer twin −1.39 ([−1.97, −0.79]). Instruction tuning is normally expected to cost benchmark macro. It did, for two of the three. We do not know yet whether that generalises — it is one run per architecture, one recipe, no seed repeats.
- The concurrency claim is a measurement on a real server, not an asymptotic argument. Constant-state papers state O(1) memory as a property. We ran a public chat service on one consumer card and counted seats admitted from real free VRAM — and we publish the two caveats that cut against the number, including that a production server with paged KV would close much of the gap on short chats.
- We publish where our own measurements come out negative. The limitations section below is longer than the results section, and includes gates our current build fails.
What we do not claim, and would like to fix. We have not run a
budget-matched head-to-head against Mamba or RWKV. We ran state-spaces/mamba-1.4b-hf through
our own harness and it scores 56.40 macro against our 45.24 —
but it is a finished public release on a far larger budget, so that number answers a different question
than the one we are asking. A matched-budget comparison inside the constant-state family is exactly the
experiment we cannot afford on our own, and it is near the top of the list of things we would run with
compute support.
What it is for. Two situations, both of them about cost rather than about answer quality:
- Many sessions at once. When a service holds thousands of live conversations, the dominant memory cost in a transformer deployment is per-session KV, and it grows with every turn. Here it is a constant. We measured what that is worth on one consumer GPU: 727 parked Z-Next sessions against 14 for a same-size transformer twin under an identical memory budget.
- Long inputs without a memory bill. Feeding the model 31,926 tokens instead of 952 moved the per-session state by zero bytes and the transient working memory from 247 MiB to 266 MiB. The 32k input was consumed in 4.8 seconds.
What it is not. This is a 1B-parameter model trained on 10B tokens. It hallucinates the way a model of that size and budget hallucinates, it does not do multi-hop reasoning, and it is not a product. The claim we are making is about the economics of the mechanism, not about answer quality. The limitations section is long on purpose.
The training run, in full
| Item | Measured |
|---|---|
| Parameters | 947,153,897 |
| Tokens seen | 10,000,007,168 (10.00B) |
| Wall-clock | 29.97 h |
| Hardware | 4 × NVIDIA H200, one node |
| Compute | 119.9 H200-GPU-hours |
| Sustained throughput | ~92,700 tok/s |
| Tokens per GPU-hour | 83.4 M |
| Loss (EMA) | 11.52 → ≈2.78 |
| Telemetry records | 15,258 |
| NaN / inf events | 0 |
| Divergence spikes after warm-up | 0 |
| Restarts / checkpoint rollbacks | 0 |
The final EMA reading is 2.7824; across the last twenty telemetry records it oscillates between 2.750 and 2.784, so we write ≈2.78 rather than quoting a single record. Marginal throughput over the 20–100% span of the run was 92,989 tok/s and 92,713 tok/s over the last 10% — throughput moved by less than 0.4% across thirty hours, so there is no thermal or memory-pressure decay hiding in the average. A pre-launch health probe on the four cards returned 744.4 / 745.2 / 745.3 / 747.3 ms per step: 0.4% spread, no sick card propping up the run.
Corpus
10.33B tokens, from two publicly addressable sources: HuggingFaceFW/fineweb-edu
(sample-100BT) and English Wikipedia (wikimedia/wikipedia, 20231101.en). No synthetic data
was used, by decision. The corpus is English, and every test on this page is English.
Honest note on the ratio: the model saw 10.00B tokens from a 10.33B-token corpus, but training windows were sampled at random with replacement, so this is not "one epoch" and the expected corpus coverage is roughly 62%. What the ratio does say is that the result does not rest on volume repetition.
Instruction tuning, and what it cost
The instruction-tuned checkpoint benchmarked below saw 3.3 million tokens of instruction data — 0.033% of the pre-training budget — in 60 minutes on a single H200. A later chat-tuned build (21 August, the one the demo is configured for) took 69.2 minutes on one H200 and cost ~$12 all-in, actually billed (about $5.30 of that is the training itself). That $12 is the only real invoice figure we have.
Pre-training rental cost is a derivation, not an invoice: 29.97 measured hours × the documented $18.36/h list rate for the 4×H200 node ≈ $550. We have no balance-before/balance-after record for that run, so we label it as arithmetic. For calibration: our earlier 1B run on the same corpus and hardware was estimated at $520–620 (also a projection), and the pre-run estimate for this one was $417 / 23 hours — the real run came in about 30% longer than we predicted. The figure we would rather you use is the one that does not move with rental prices: ~120 H200-GPU-hours.
A constant-size session turns concurrency into a different bill
In an attention-based deployment, each live conversation owns a KV cache that grows with every token. That cache, not the weights, is what caps how many conversations one GPU can hold. In Z-Next the per-session footprint is a constant, and it is enforced in code: loading a checkpoint fails the self-test if the session state is not exactly 5,480,448 bytes (= 5.227 MiB, fp32), and a separate check verifies it does not grow after a turn.
We measured the difference on a single 16 GB consumer card (NVIDIA RTX 5070 Ti), running Z-Next and a same-size transformer twin side by side on the same card at the same time, with the measured free VRAM split half and half. Both models are fp32, so the comparison is symmetric in numeric precision. Capacity is read from the card at start-up (not from a spec sheet); if something else is using the GPU, the advertised seat count is reduced and a warning is printed, and if a CUDA OOM is ever observed at runtime the advertised number is lowered to the observed reality.
| Z-Next | Transformer twin | Ratio | |
|---|---|---|---|
| Weights | 3,651 MiB | 3,830 MiB | — |
| Per-session memory | 5.227 MiB, constant | 256.0 MiB, pre-allocated | 49× |
| Grows with context? | no | ceiling reserved up front | — |
| Sessions held in the half-budget | 727 | 14 | 51.9× |
| Same card, Z-Next alone | 2,220 | — | — |
| Same card with 5,151 MiB of foreign load | 258 | 4 | 64× |
Arithmetic, in the open: (7,804 − 3,651 − 350) ÷ 5.227 = 727 · (7,804 − 3,830 − 350) ÷ 256 = 14. The 727/14 pair reproduced identically across four separate engine start-ups; the 2,220 figure across ten. The 256.0 MiB is what the twin reserves for a 1,024-token ceiling at session open — it allocates the whole ceiling at once rather than growing into it.
What that ratio does not mean — stated by us, before you ask
Seats are an admission limit, not throughput. 727 seats means 727 parked conversations whose state is resident in memory. It does not mean 727 simultaneous token streams: generation on a single GPU is serialized. Anyone quoting this as a throughput number, including us, would be overstating it.
The two sides were not stressed the same way. The twin's 14 was driven into the wall end-to-end — we opened capacity+3 sessions, the overflow got HTTP 503, and the advertised ceiling held. Z-Next's 727 is a quotient of measured free budget over measured constant cost; the largest number of Z-Next sessions we actually opened live is 20, where VRAM consumption tracked the linear prediction within ±1 MiB.
The biggest lever is the twin's context ceiling, and a better-engineered twin closes much of the gap. See below.
- The ratio is near-linear in the twin's declared ceiling. At the same budget: 256 tokens → ~13×, 1,024 tokens → 52× (this is the measured one), 2,048 → ~104×, 4,096 → ~242×. Everything in that list except the 52× is arithmetic extrapolation from the measured per-token cost, not a measurement.
- This twin is a plain implementation that reserves the ceiling up front. Production serving stacks with paged or block-wise KV allocate as the conversation grows. On short conversations such a server holds far more sessions than 14, and the ratio drops substantially. This is the strongest argument against the headline number and we would rather publish it than have it discovered.
- Both sides are fp32. Moving both to bf16 roughly preserves the ratio while raising seat counts on both sides. Quantizing only the twin's KV to int8/fp8 would cut the ratio by 2–4×.
- On the one tighter-card configuration we measured, the ratio went up rather than down (258 vs 4 = 64× with 5,151 MiB of foreign load on the card), because the twin's 256 MiB granule rounds the leftover budget more coarsely. That is a single observation, not a law: the ratio is a quotient of two integer divisions and it moves in steps, so it can jump either way as free memory changes. We have not measured a larger card at all, so we make no claim about one.
Long inputs are free in memory — this one is a direct measurement, not a projection
| Input tokens | Prefill | Prefill rate | Transient VRAM | Session state |
|---|---|---|---|---|
| 952 | 1.3 s | 723 tok/s | 247 MiB | 5.227 MiB |
| 4,012 | 1.1 s | 3,638 tok/s | 266 MiB | 5.227 MiB |
| 9,928 | 1.6 s | 6,357 tok/s | 266 MiB | 5.227 MiB |
| 19,924 | 2.9 s | 6,922 tok/s | 266 MiB | 5.227 MiB |
| 31,926 | 4.8 s | 6,699 tok/s | 266 MiB | 5.227 MiB |
The 266 MiB is shared, transient working memory during generation — it is not per seat, and it is the reason 350 MiB is withheld from each half-budget above. The session state column is the per-seat cost, and it is bit-for-bit identical across a 33× range of input length. Caveat that belongs right here: being able to ingest 32k tokens cheaply is not the same as being able to use them. See Limitations, items 1–3 — retrieval from a long natural document currently fails a few thousand tokens in.
Ten tasks, full test sets, zero-shot, one seed, pure fp32
Every row below was produced by our own harness on our own hardware — including the Pythia rows, which are not copied from published tables. No subsampling: the full test or validation split of each task was scored (LAMBADA 5,153 · HellaSwag 10,042 · PIQA 1,838 · WinoGrande 1,267 · ARC-Easy 2,376 · ARC-Challenge 1,172 · OpenBookQA 500 · SciQ 1,000 · BoolQ 3,270 · MMLU 14,042). Single seed (1234), pure fp32 with TF32 disabled, 1,024-token context ceiling applied to every model.
Metric contract, applied identically to every row: length-normalized accuracy for HellaSwag, PIQA, ARC-Challenge, OpenBookQA and SciQ; plain accuracy for LAMBADA, WinoGrande, ARC-Easy, BoolQ and MMLU. Under the alternative contract (plain accuracy on PIQA) no macro moves by more than 0.14 p.p. and no verdict on this page changes. We recomputed every macro from the raw per-item outputs rather than transcribing them.
↔ table scrolls horizontally on narrow screens
| Model | Tokens | Params | LAMB | HELLA | PIQA | WINO | ARC-E | ARC-C | OBQA | SCIQ | BOOLQ | MMLU | MACRO |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Z-Next 1B — instruction-tuned | 10.00B | 947M | 30.5 | 41.8 | 66.9 | 50.5 | 56.6 | 30.6 | 32.8 | 68.8 | 55.6 | 23.4 | 45.75 |
| Z-Next 1B — base | 10.00B | 947M | 30.5 | 40.1 | 67.0 | 51.6 | 57.2 | 29.0 | 33.0 | 65.9 | 54.5 | 23.5 | 45.24 |
| Previous-gen Z — base | 10.32B | ~1B | 34.3 | 40.6 | 65.9 | 50.7 | 55.6 | 28.8 | 33.0 | 67.0 | 57.4 | 24.0 | 45.73 |
| Previous-gen Z — same SFT recipe | 10.32B | ~1B | 31.3 | 42.2 | 67.0 | 50.5 | 55.1 | 30.2 | 32.8 | 68.5 | 51.4 | 24.0 | 45.30 |
| Same-data transformer twin — base | 10.32B | 937M | 31.0 | 38.2 | 63.5 | 53.2 | 52.8 | 26.1 | 31.8 | 64.6 | 59.8 | 25.1 | 44.62 |
| Same-data transformer twin — same SFT recipe | 10.32B | 937M | 22.5 | 39.9 | 62.5 | 53.7 | 51.2 | 29.4 | 29.2 | 64.7 | 54.2 | 24.9 | 43.24 |
| Pythia-1.4B @ step 5000 | 10.49B | 1,415M | 42.2 | 32.5 | 62.2 | 50.6 | 46.4 | 23.9 | 28.0 | 69.8 | 56.1 | 23.2 | 43.49 |
| Pythia-1B @ step 5000 | 10.49B | 1,012M | 36.6 | 31.8 | 62.4 | 53.2 | 44.6 | 22.7 | 26.6 | 69.4 | 53.7 | 23.0 | 42.40 |
Macro uncertainty for a single model is 1σ ≈ 0.38 p.p. (binomial, from the item counts). For a difference between two models that figure is the wrong tool, so every comparison below uses an item-level paired bootstrap (4,000 resamples) with a 95% confidence interval.
Every comparison, with its confidence interval
| Comparison | Δ macro | 95% CI | Verdict |
|---|---|---|---|
| Z-Next base vs Pythia-1B @10.49B | +2.83 | [+2.10, +3.57] | ahead |
| Z-Next base vs Pythia-1.4B @10.49B | +1.74 | [+1.03, +2.43] | ahead |
| Z-Next base vs same-data twin base | +0.62 | [−0.06, +1.28] | equal — CI spans zero |
| Z-Next instruction-tuned vs twin + same SFT | +2.51 | [+1.79, +3.25] | ahead |
| Z-Next instruction-tuned vs twin base (the twin's own best checkpoint) | +1.13 | [+0.42, +1.81] | ahead |
| Z-Next instruction-tuned vs previous-gen Z + same SFT | +0.45 | [−0.22, +1.11] | equal — CI spans zero |
| Z-Next at 74% of budget vs finished twin | +0.06 | [−0.59, +0.73] | equal — CI spans zero |
| Effect of SFT on Z-Next | +0.51 | [+0.04, +0.98] | small, positive, barely clears zero |
| Effect of the same SFT on the transformer twin | −1.39 | [−1.97, −0.79] | clear regression |
| Effect of the same SFT on previous-gen Z | −0.43 | [−0.89, +0.04] | regression, borderline |
1. Against budget-matched external baselines the base model is ahead by 2.83 and 1.74 points, with confidence intervals well clear of zero, while having seen 4.7% fewer tokens and carrying 6.4% / 33% fewer parameters.
2. The same instruction-tuning recipe, run on three architectures, moved the macro −1.39 on the transformer twin, −0.43 on our previous generation, and +0.51 on Z-Next. We are not appealing to a literature generalization about alignment tax here; these are three controls we ran ourselves. It is the first direct measurement we have of "robustness to instruction tuning" differing by architecture. Its evidential weight is small and we will not inflate it: one recipe, one tuning run per architecture, no seed repeats, n=3 architectures. It is a reason to run the experiment properly, not a finished result.
3. At 74% of its budget (7.42B tokens) the model already matched the finished twin (Δ = +0.06, CI spans zero) and was ahead of both Pythia checkpoints (+2.28 and +1.19, both CIs clear of zero).
What we refuse to claim, twice over. First: that Z-Next beats our own previous generation. That gap is +0.45 with a CI of [−0.22, +1.11]. The honest word is equal, and on two other measures below the previous generation is actually better. Second: that the base model beats the twin's base. That gap is +0.62, CI [−0.06, +1.28] — it also contains zero. The only architecture comparison on this page that clears its interval against the twin is the one run after the same instruction-tuning recipe, and a large part of that margin is the twin losing 1.39 points to the recipe rather than us gaining.
Perplexity
Word-level perplexity and bits-per-byte are tokenizer-independent, so these are the numbers that may legitimately be compared against a model with a different vocabulary:
| Model | Word PPL | Bits/byte |
|---|---|---|
| Z-Next @ 9.39B tokens (94% of budget) | 29.88 | 0.9162 |
| Same-data transformer twin (finished) | 29.76 | 0.9151 |
| Previous-gen Z (finished) | 29.66 | 0.9142 |
| Pythia-1.4B @10.49B | 39.59 | 0.9921 |
| Pythia-1B @10.49B | 41.20 | 1.0029 |
Version caveat, and it matters: WikiText-103 was never run on the finished 10B model. The furthest checkpoint we measured is 94% of budget, and it is the one in the table; the two in-house baselines are finished models. Against them we are marginally behind (+0.4% word PPL) — an earlier reading at 74% showed a real gap of 32.57, and it closed by 94%. Against the budget-matched external baselines the margin is large and the comparison is methodologically sound.
| Model | LAMBADA PPL |
|---|---|
| Previous-gen Z (finished) | 31.59 |
| Z-Next base (finished) | 42.54 |
| Same-data transformer twin (finished) | 62.68 |
The Pythia models are deliberately absent from this table: token-level perplexity across different vocabularies is not a comparison, and publishing one would be a mistake in either direction. Within the shared-tokenizer family, Z-Next assigns markedly better probability than the transformer twin (42.54 vs 62.68) and markedly worse than our own previous generation (31.59).
The fairness gates, and the places the setup works against us
A benchmark comparing two architectures is only worth reading if the harness cannot silently favour one of them. Four checks are recorded on every run, per model, and a run that fails any of them is rejected rather than reported.
| Gate | What it rules out | Threshold | Measured range |
|---|---|---|---|
| G1 — padding immunity | right-side padding shifting the logits of earlier positions, i.e. batching helping or hurting a model | < 1e-2 | 0.00 – 2.37e-04 |
| G2 — narrow-output identity | the scoring path disagreeing with the full forward pass | < 1e-3 | 1.43e-05 – 1.43e-04 |
| G3 — context ceiling | one model getting more context than another | 1,024, all models | see note |
| G4 — fast-path parity | our accelerated inference path drifting from the reference computation | < 1e-2 | 9.16e-05 – 2.75e-04 |
G4 is not universal and we will not pretend it is. It only asks a question that exists
for models that have our accelerated inference path; for the transformer twin and for Pythia it is
recorded as not-applicable. So the accurate sentence is "three gates passed on every model, and the
fourth — which only applies to us — passed on us," not "all four gates passed everywhere."
G3 note: zero items were truncated for our models and for the twin; 2 items out of
40,660 were truncated for each Pythia row and 22 for TinyLlama. Negligible, but "no item was ever
truncated" would be false.
Four places the setup is tilted against us — stated plainly
① We saw the fewest tokens of anyone in the table
Z-Next: 10.00B. The transformer twin and previous-gen Z: 10.32B (+3.2%). Pythia: 10.49B (+4.9%). Calling this an "equal budget" would be wrong; the accurate statement is that everyone else got more.
② We are the smaller model against the external baselines
947M parameters against Pythia-1B's 1,012M (−6.4%) and Pythia-1.4B's 1,415M (−33%).
③ The context ceiling is set by the twin, and it costs us our main advantage
Every model in the benchmark is capped at 1,024 tokens, because that is the twin's position-table limit and the cap must be identical for all. Which means Z-Next's constant-state long-context behaviour — the reason the architecture exists — is not exercised at all in this table.
④ Pythia was trained on different data
What we equalized is the token budget and the measurement protocol, not the corpus (Pythia was trained on the Pile). The comparison that is data-controlled is the one against the transformer twin and previous-gen Z: identical corpus, identical tokenizer, identical trainer.
Two models we measured and did not put in the table
Fully-trained public 1B-class models score well above us, and we ran them ourselves
The same harness, same seed, same 1,024-token ceiling and the same gates were also run on
TinyLlama/TinyLlama_v1.1 (1,100M params) and state-spaces/mamba-1.4b-hf
(1,372M params). Under the identical metric contract they score 48.55 and
56.40 macro — that is 3.3 and 11.2 points ahead of our base model.
They are not in the main table because neither dimension is matched. Mamba-1.4B carries
1,372M parameters against our 947M — 45% more — and it is a finished
public release whose pre-training budget is far larger than our 10.0B tokens. Its own model card states no
token count, and we did not verify one ourselves, so we quote no number for it. Two variables move at once;
that is not an architecture comparison.
For contrast, the one place where we do have budget-matched public baselines is on the transformer side: Pythia-1B and Pythia-1.4B at ~10B tokens score 42.40 and 43.49 through the same harness — both below us, and the 1.4B is 49% larger than we are. Inside the constant-state family, no budget-matched comparison exists at all, ours or anyone's. “Would a Mamba trained on the same 10B tokens beat this?” is, as far as we can tell, an open question — and one we would like to close rather than win.
We are publishing this rather than leaving it out, for two reasons. First, a table that silently omits the strongest models the harness measured is a cooked table. Second, Mamba is the closest published member of the constant-state family we say we belong to, and 56.40 is the honest answer to “how do you compare to it?”: at 1B scale and matched budget we do not have a quality argument against a model trained on a far larger budget. What we have is a matched-budget comparison and a memory profile. That is the whole claim.
Two caveats that go the other way, or sideways
The twin is not a shape-for-shape clone
Z-Next and the transformer twin are not identical in depth/width profile. What the comparison isolates is therefore "architecture + shape" as a package, not architecture alone. We give no numbers here because depth and width are the parts of the design we do not publish; you should read the twin comparison with that discount applied.
Two measurement sessions, not one
The five base/SFT rows were measured on 19 August in one session; the baseline rows on 18 August in a separate session. Same harness, same seed (1234), pure fp32, TF32 off — but not the same run.
Loss as a function of how much context the model was given
Identical windows for all three models (seed 4242, 16 independent windows per length, same end points with different look-back). Only the last 256 tokens are scored, which turns the measurement into one clean question: how much easier did having T tokens of context make the final tokens? Lower is better; units are nats.
Two things this table is not. First, the windows are drawn from the same corpus all three models were trained on — it is not a held-out set. The text is identical for all three, so the comparison between them stays symmetric, but none of these numbers is a generalization measurement, and the three models did not see identical fractions of that corpus (10.00B draws for us against 10.32B for the other two). Second, we did not compute a confidence interval here: each cell is 16 windows × 256 scored tokens, and the differences we are pointing at are 0.02–0.05 nats. Treat the ordering as consistent-across-five-lengths evidence, not as an interval-tested result.
| Context T | Z-Next @ 74% of budget | Transformer twin (finished) | Previous-gen Z (finished) |
|---|---|---|---|
| 512 | 2.8084 | 2.8558 | 2.7577 |
| 1,024 | 2.7750 | 2.8036 | 2.7156 |
| 2,048 | 2.7694 | 2.7904 | 2.7133 |
| 4,096 | 2.7656 | 2.7866 | 2.7117 |
| 8,192 | 2.7656 | 2.7873 | 2.7130 |
| context gain 512→8,192 | −0.043 | −0.069 | −0.045 |
Read this both ways. At 74% of its budget Z-Next is below the finished transformer twin at every one of the five lengths — by 0.022 nats at 8,192. And at every one of those same five lengths our own previous generation is better than both. We are showing that column rather than quietly dropping it; a table that only contained the twin would have been a more flattering and less honest table.
Why the 74% checkpoint: when the finished model was later re-measured on the same windows in fp32 it returned 2.7590 / 2.7189 / 2.7142 / 2.7107 / 2.7093 with a context gain of −0.0497 nats — better than its own 74% self at every length. But the two baselines were not re-run in that session, so those numbers are not a like-for-like row and we do not put them in the table above.
One further measured point: instruction tuning made raw-corpus perplexity ~17% worse (2.7438 → 2.9013 nats; PPL 15.5 → 18.2), which is the expected distribution shift — but the context gain survived it and slightly grew (−0.0497 → −0.0564 nats). The ability to exploit long context was not traded away for instruction following.
What this model cannot do, and where our own measurements come out negative
This section is longer than the results section. That is deliberate: we would rather you find the holes here than in a call with us. Nothing below is softened, and several items directly undercut the more exciting framings of the numbers above.
1 · Long-range binding in free text is not solved. This is our most important negative result.
We built a neutral probe nobody was trained on: plain English sentences embedding four project codes into real corpus text, then a forced-choice question about one of them. Chance is 25%; n = 216 per model (3 context lengths × 3 depths × 24 items).
Base model 0.306 · instruction-tuned 0.301 · difference −0.005, p = 0.92. No effect whatsoever. The depth breakdown is worse: all three models we tested work only in roughly the last 15% of the context (0.58–0.92 there) and score below chance at head and middle depth.
So: "Z-Next solved recall" is a sentence we will not write. The most we will claim is that interference resistance in structured binding scales with curriculum — see item 2.
2 · The large gain we did measure is format-specific, and the curve still falls
In a structured setup (32 key→value records inside an 8,192-token window, then a variable amount of intervening plain text, then one key is asked; chance ≈ 1/64) instruction tuning moved accuracy over the six unsaturated points from 0.292 to 0.681 (p = 4.1e-11), and at a 7,007-token gap from 0.083 to 0.458 — a 5.5× improvement (p = 0.0035), with no architectural change, from 3.3M tokens of instruction data.
The honest framing of the same curve: 100% correct at a 280-token gap, ~46% at a 7,000-token gap, monotonically decreasing throughout. And per item 1, none of it transferred to neutral natural language.
3 · On the live demo, retrieval from a natural document dies after a few thousand tokens
Measured on the same pipeline the public demo runs, on the 20 August chat-tuned build, with non-degenerate filler text: a ~500-token document → correct; ~2,700 tokens → wrong; ~8,000 → wrong; ~18,000 → wrong. You can reproduce this yourself in the demo, and we would rather you did.
Our internal reading is that this is a curriculum limit rather than an architectural one (the long-gap portion of the last tuning run was only a few hundred examples). We flag that clearly as a hypothesis, not a measurement.
4 · The capacity improvement is below our own significance threshold
How many records the model can hold at once: pooled over four settings (n = 96) instruction tuning moved 0.385 → 0.521, z = 1.88, p = 0.059. The direction is consistent on 4 of 4 points, but this does not clear the bar. It is a signal, not a verdict, and n needs to grow before it becomes one.
5 · Composition / multi-hop reasoning is essentially absent
Two-hop questions: 1–3 correct out of 24 when asked directly, 0 out of 24 when chained — on the base model and both tuned variants alike. This is an open capability at 1B scale, not something we have addressed.
6 · We are not ahead of our own previous generation
Macro: +0.45 with a 95% CI of [−0.22, +1.11] — equal. Token-level LAMBADA perplexity: previous generation better (31.59 vs 42.54). Long-context loss: previous generation better at all five lengths. Whatever else this release is, it is not a clean generational win over ourselves.
7 · Specific tasks where we lose, despite leading on macro
WinoGrande: 51.6 vs the transformer twin's 53.2 (−1.6 p.p.; the gap was −3.6 at the 74% checkpoint). This is a recall-flavoured task and the shortfall is a known signature of the same weakness as items 1–3.
LAMBADA accuracy: 30.5 vs Pythia-1B's 36.6 and Pythia-1.4B's 42.2 — 6.1 and 11.7 points behind, on a benchmark where we lead the macro. We keep it in the table.
8 · One of the ten tasks discriminates nothing at this scale
MMLU lands between 22.97 and 25.11 for every 1B-class model we measured, against a chance level of 25.0. It contributes a tenth of the macro and separates nobody. We report it because dropping an uninformative task after seeing the results is how benchmark tables get cooked.
9 · Instruction following is weak in a specific, measurable way
Asked to answer in a mandated output shape ([CODE: <answer>]), the model produced
the required format in none of 5 seeds × 5 decoding settings — while still
writing the correct answer somewhere in the surrounding prose in 80% of those runs. It answers and
ignores the requested form. That is a format-following failure, distinct from recall, and it is real.
Which checkpoint: measured on the 20 August chat-tuned build, not on the instruction-tuned checkpoint benchmarked in section 3, and not re-run on the others. We cannot tell you from this whether it is a property of the line or of that one build.
10 · Tool use inside a long session is unreliable — and it moved a lot between builds
On our 20 August chat-tuned build, end-to-end tool success is 0.033–0.067 against a pre-registered threshold of 0.40, and 0.000 on the worst seed; the call rate is 0.733 against a threshold of 0.85. In a fresh context, end-to-end success on that same build is 0.600 — so on that build the long session, not the task, was the failure point. Read this paragraph together with the correction below: it did not stay true of the next build.
Root cause is measured, not guessed: values leak across turns, at 0.089–0.097 against a pre-registered leakage threshold of 0.06–0.07. That gate failed. The previous build was twice as bad (0.18–0.20); halving it was not enough to pass.
Correction — and it cuts in our favour, which is why we would rather write it than leave a conveniently pessimistic number standing. The 21 August chat-tuned build has since been through the identical gate suite (15 seeds, same temperature, same mixed-session harness) and moved these numbers a long way: mid-session end-to-end tool success 0.567 — clearing the 0.40 gate, with the worst of 15 seeds at 0.500 — call rate 1.000, chat recall 0.944, refusal rate 0.031. So "the session destroys the capability" was true of one build and is not a property of the architecture. What still fails on that newer build: value leakage across turns 0.067 against a 0.060 threshold; topic-holding inside a long mixed session 0.658 against 0.680; and the worst of 15 seeds drops chat recall to 0.667 against a 0.700 floor. 3 of 17 pre-registered gates fail. Thresholds were fixed before the run and not moved afterwards.
11 · Chat quality is not product-grade, and hallucination is what you would expect at this size
Representative failures from our own logs: the Moon's orbital period given as "21 days"; "the first three primes are 1, 2, 3"; an arithmetic answer of 380,793 where the correct value is 380,794. A 1B model trained on 10B tokens produces this. The claim of this work is the economics of the mechanism, not the quality of the answers.
12 · The tables, the chat measurements and the live demo are three different checkpoints
Everything in section 3 belongs to the base and instruction-tuned checkpoints. The chat and tool-use numbers in items 9 and 11, and the first half of item 10, belong to our 20 August chat-tuned build; the corrected figures in item 10 belong to the 21 August one. The public demo serves one of the later chat-tuned builds — the 20 or the 21 August one, selected when the engine is started, so we cannot promise you in advance which one answers you. The 21 August build's chat gates have been measured and are reported in item 10; what has never been run, on either chat-tuned build, is the classic benchmark suite. So the 45.8 in section 3 is not the score of the model you are talking to, and no score for that model exists. Do not read the demo's behaviour as evidence for the tables, or the tables as a promise about the demo. When that benchmark is run, the number will be published whichever way it goes.
13 · Scope of the evidence
One pre-training run, one seed, no repeat run at this budget. Macro uncertainty for a single model is 1σ ≈ 0.38 p.p. The corpus and every test are English. Everything here is at ~1B parameters and ~10B tokens: nothing on this page tells you how any of it behaves at 7B, 70B, or at a trillion-token budget. Establishing that is exactly what we are looking for support to do.
A live public demo, running on one consumer GPU in our office
In its comparison mode it serves Z-Next and the same-size transformer twin side by side on the same card, at the same time, on the same half-and-half memory budget, and the page reads a live status panel showing each side's session count and its capacity. That is the concurrency table in section 2, running rather than tabulated. The engine can also be run with the twin disabled, in which case only the Z-Next side is shown.
Before you open it, four things:
- This is a 1B research model. It will make things up. See limitation 11.
- The demo runs a later chat-tuned checkpoint than the one benchmarked above, picked from our two most recent chat builds when the engine is started. No chat-tuned build of this model has ever been run through the classic benchmark suite, so the 45.8 in section 3 is not its score and it has no score of its own. Its chat gates have been measured, and 3 of 17 of them fail — see limitations 10 and 12.
- It runs on one 16 GB consumer card behind a home connection, gated by CAPTCHA and proof-of-work, and it is off by default — it is switched on for demonstration windows. If it is dark when you arrive, that is why; write to us and we will turn it on.
- The most interesting thing you can do with it is try to break the retrieval. Paste a long document with a fact buried in it. Around a page it works; a few thousand tokens in it stops. That is limitation 3, and we would rather you confirm it than take our word for it.
What we are looking for
The plan, in three stages, each with a stop condition written before the run. A compute grant is easier to judge when it buys a specific experiment rather than a direction:
| stage | run | compute | gate |
|---|---|---|---|
| 1 | 1B on 60B tokens — 6× our current budget, everything else held fixed | ≈720 GPU-h | macro ≥ 47.5, twin lead preserved, no long-context regression. If it fails, we stop. |
| 2 | 3B on 200B tokens | ≈7,600 GPU-h | must beat a budget-matched 3B transformer twin and move a neutral-prose needle test off chance — the one thing our 1B could not do (p=0.92) |
| 3 | 7B + tool use, coding and agentic work, shipped | ≈17,700 GPU-h | only if stage 2 passes |
We are asking for stage 1 only: ≈720 GPU-hours. It answers one question — what six times the token budget does to this architecture — and we publish the result either way. The compute figures are extrapolated from our measured run (947M parameters, 10.0B tokens, 29.97 h on 4×H200 ≈ 120 GPU-hours) by scaling throughput with parameter count. They are estimates, not measurements, and they will drift with sequence length, parallelism and hardware generation.
The result on this page has a hard ceiling built into it, and we said so in limitation 13: everything was measured at 947M parameters and 10B tokens. The question that matters — whether a constant-state architecture holds its parity, and keeps its memory economics, at 7B and beyond, and at a serious token budget — is not a question we can answer with 120 GPU-hours.
We are a small team. We built the architecture, the trainer, the benchmark harness with its fairness gates, the serving engine and the public demo, and we ran the entire 10B pre-training on about 120 H200-GPU-hours — the one cost figure here that is not sensitive to rental prices, and the one we would rather be judged on than the derived dollar number in section 1. We know how to spend compute carefully and how to report what it produced, including when the answer is no. What we need now is scale.
Concretely, we are looking for
- Compute — a grant, credits, or a partnership that puts a multi-node cluster and a serious token budget behind a scale-up run.
- Research collaborators — particularly people working on constant-state and state-space sequence models, on long-context evaluation methodology, and on serving economics. The long-range binding result in limitation 1 is an open problem and we would like to work on it with people who find it interesting.
- Deployment partners — if your cost structure is dominated by holding a large number of concurrent sessions rather than by peak single-stream quality, the economics in section 2 are worth a conversation. Be aware of what section 5 says about current quality.
- Adversarial reviewers — if you think one of these numbers is wrong, or that a comparison here flatters us, tell us. Several of the corrections on this page came from exactly that kind of pushback internally, and we would rather it happened before publication than after.
How to reach us
Email — the fastest route: [email protected]. Put "Z-Next" in the subject line and it reaches the people who built it.
What we can share under NDA: the full measurement chain behind every table on this page — raw per-item benchmark outputs, training telemetry, the fairness-gate records, the engine logs behind the concurrency numbers, and the negative results in full. We are glad to walk a technical reviewer through all of it.
What we do not share: the mechanism — how the session state is maintained, the model's shape, and the training recipe. A method patent application is in preparation. Everything on this page was written so that it stands on measurements alone, without you needing any of that.
If you just want to look first: the demo is public, and our other measurement write-ups are in Research Notes.
Email us about Z-Next Try the demo Research Notes
Every figure on this page comes from a run we performed and can show. Where a number belongs to an intermediate checkpoint rather than the finished model, it is labelled as such. Where a comparison is uneven, the direction of the unevenness is stated. Where a result came out negative, it is in section 5 rather than absent. If you find something here that does not meet that standard, we want to hear about it. Measured 17–21 August 2026.