Z-Next 1B

What you can check yourself

• Open weights: VAN-1B, the transformer trained on the same corpus. Download it from Hugging Face (Apache-2.0) and run the ten benchmarks of the results table on it yourself. Its rows there are the baseline of every comparison with Z-Next on this page, so you can check that baseline without asking us. Its model card is on this site.
• The live demo. When it is switched on, both models answer the same prompt side by side at tetracta.ai/zchat, and the state size is shown.
• The live village. ztown.tetracta.ai, for visitors aged 18 and over.
• The intervals. Each difference in the results table carries a 95% paired-bootstrap interval.
• The corrections. The 10 Nov 2026 changelog entry says what was corrected on the Hugging Face card.

Informal tests in the demo. The transformer baseline in the demo has a context limit of 1,024 tokens. A longer note does not fit, so its session reports a full window and resets. Z-Next has no such limit, and its session state did not change in size in any of these runs. The comparison shows constant state. It does not show better recall.

We pasted a note into both models that states "the override code for the cold room is 7391", surrounded by filler text, and then asked for the code. With the code at the start of the note, Z-Next answered correctly at every length we tried, from about 1,000 to about 22,000 words. With the code in the middle it was right at 1,000, 4,000 and 16,000 words, and wrong on the first run at 8,000.

Some facts came back and others did not. The code came back in most runs. A cat's name and a ferry time placed in the same kind of note often did not. In a chat of twelve messages that gave a name, a code and a cat's name in the first message, both models returned only the name. These are single runs in a demo, not a benchmark. Measured recall in free-form language is near chance (see Evaluation).

What it is for

Z-Next holds each conversation in a state of fixed size, however long the conversation runs. A fixed state helps in two places: many long-running sessions on one GPU (game characters are the case we run), and machines with a fixed memory budget. Long documents fit in the state too, but recall over them is near chance at this size (see Limitations).

Measured today. The live village: 25 residents on one gaming GPU, with a median decision time of 1.07 s.

Not shown yet. Serving many sessions at once, any scale beyond 1B, and a matched comparison with Mamba and RWKV.

Our hypothesis. Trained at larger scale, Z-Next may reason better than a conventional model of the same size. Today's 1B model fails two-step questions (0 of 24 on our test), so this is a hypothesis about larger scale, not a result.

Model details

In a transformer, the memory a conversation uses grows with its length, because the KV cache keeps keys and values for every past token. Z-Next keeps no such cache. Everything it carries from one token to the next sits in a state of fixed size. Mamba and RWKV are the best-known published models that also keep a fixed state, and that is what Z-Next shares with them. It is not a variant of either. It has no attention layer, and the way it writes to its state and the way it learns are our own design. We explain both under NDA.

Instruction-tuned variants

On top of the base model we trained four instruction-tuned variants, each on one H200 GPU. This page refers to them by role.

Intended use

Primary uses

• Research on attention-free, fixed-state language models at the 1B scale.
• Measuring state size per conversation and model behaviour on your own hardware and workload, in a joint evaluation under NDA.
• Technical evaluation by research partners and by clients considering a custom model.
• Hands-on inspection through the public demo, when it is up.
• Choosing actions for characters in Z-Town, from the actions the game allows (see the Z-Town case, or enter the live village, for visitors aged 18 and over). The system card covers both deployments.

Out of scope

• Production deployment of any kind.
• Factual question answering, or any use where the output has to be correct.
• Advice of any kind: medical, legal, financial, safety or otherwise.
• Any decision about people, such as hiring, credit, housing, education or access to services.
• Safety-critical use.
• Any language other than English.
• Finding facts in long documents. Long-range recall is near chance (see Evaluation).

Training data

Pre-training corpus

Two public English sources: FineWeb-Edu (the sample-100BT subset) and English Wikipedia (the 20231101.en dump). Our data pipeline applies adult-content and spam filters to them. The corpus file holds 10.33 billion tokens; the model consumed 10.0 billion. Training windows were sampled from the corpus file, so we do not claim one epoch or no repetition. The ratio between the two sources was a collection target and was never measured, so we quote none.

Every 1B model we train uses this same corpus file and tokenizer, including the transformer twin VAN-1B.

Program-generated material

The corpus file contains no synthetic text. During pre-training, however, a small share of Z-Next's pre-training windows also carried program-generated material that the transformer did not see. That material is synthetic data. VAN-1B saw plain language modelling only. The difference bears on every comparison between the two models; see Evaluation.

Instruction data

The benchmark variant was tuned on 12,546 examples (3.31 million tokens), equal to 0.033% of the pre-training token budget. The chat variants were tuned on chat and tool-use examples. Provenance of the instruction data is only partly documented: part of the base instruction set was machine-generated or machine-translated and its origin cannot be verified. For the examples added for the product variant, provenance had not been audited as of 11 Nov 2026.

Training procedure

Pre-training is next-token prediction on the corpus, plus the program-generated material described under Training data. Each instruction-tuned variant was then trained on one H200; the two runs we have documented took 60.1 minutes (the benchmark variant) and 69.2 minutes. We do not publish the rest of the recipe: optimiser settings, learning rates, schedules and model shape stay private.

The run was produced with our own training pipeline, which also trained VAN-1B.

Evaluation

Where a number comes from a checkpoint other than the finished base model, the row or sentence says so.

Setup

Ten standard zero-shot tasks, full test or validation sets with no sub-sampling: LAMBADA (5,153 items), HellaSwag (10,042), PIQA (1,838), WinoGrande (1,267), ARC-Easy (2,376), ARC-Challenge (1,172), OpenBookQA (500), SciQ (1,000), BoolQ (3,270) and MMLU (14,042). Every model goes through the same harness with one metric contract: accuracy for LAMBADA, WinoGrande, ARC-Easy, BoolQ and MMLU, and normalised accuracy for the other five. Seed 1234, fp32, TF32 off, and a context ceiling of 1,024 tokens for every model. That ceiling is the twin's positional limit, so these benchmarks do not test Z-Next on long inputs at all.

The macro score is the unweighted mean of the ten tasks. For one model, the macro has a standard deviation of about 0.38 points. For a difference between two models we report a 95% confidence interval from a paired bootstrap over per-item correctness, with 4,000 resamples. Each run also records four fairness checks: padding does not change earlier outputs, restricted-output scoring matches full-vocabulary scoring, the context ceiling is the same for every model, and our optimised inference build gives the same outputs as the reference build. The first three passed for every model. The fourth exists only for our own models, and passed there.

Benchmark scores

Some of our internal records list 45.52 for the base model. We have not resolved which harness version produced that figure, so we report 45.24, the value recomputed from per-item outputs under the contract above. Fully trained public models of similar size score higher in the same harness: 56.40 for state-spaces/mamba-1.4b-hf and 48.55 for TinyLlama v1.1. The Mamba model is larger (1.37B parameters against 0.95B), and both were trained on far more than 10 billion tokens, so this is not a like-for-like comparison.

Comparison with the transformer twin

VAN-1B is a classic transformer of about the same size that we trained on the same corpus with the same tokenizer and harness, as a control. Its weights are public.

Read the twin rows with this caveat. Z-Next and VAN-1B share the corpus and tokenizer, but not an identical training recipe: a small share of Z-Next's pre-training windows also carried program-generated material that the transformer did not see. VAN-1B is also not matched to Z-Next in shape, and it saw 3.2% more tokens (10.32 billion against 10.00 billion). So the base-to-base result supports one claim: at this scale, the architecture did not cost quality. It does not show an architectural advantage. The Pythia lead is not architectural either: VAN-1B leads Pythia by a similar margin, and Pythia was trained on a different corpus (the Pile).

After instruction tuning

After the same instruction recipe, the benchmark variant leads the tuned VAN-1B (second row of the table above). Most of that gap is the twin falling. The recipe cost VAN-1B 1.39 points, and almost all of the loss sits in two tasks: LAMBADA fell by 8.50 points and BoolQ by 5.63 points. The same recipe raised Z-Next by 0.51 points. The three parts add up to the gap: 0.62 + 1.39 + 0.51 = 2.52, within rounding of 2.51. This is one recipe and one run per architecture, with no repeated seeds, so we do not know yet whether the pattern generalises.

Product variant

The product variant, which runs the ZChat demo and Z-Town, scores 45.34 on the same benchmarks: level with the base model. Before it was trained we wrote down 17 acceptance gates for chat and tool use, and the rule that all of them must pass. It passed 14, so by that rule it failed. We use it anyway: it is the best chat variant we have.

Language modelling

Validation perplexity of the base model is 15.1 at a context of 512 tokens. This is a token-level figure, so it compares only with models that use our tokenizer. VAN-1B's published perplexity of 13.56 uses an older metric and does not compare with it. Instruction tuning made raw-corpus perplexity about 17% worse, the expected shift toward chat-style text. We have not measured WikiText-103 perplexity on the finished checkpoint; the reading on the Hugging Face card comes from 94% of the token budget and is labelled that way there.

Long-context recall

On a format-neutral, natural-language needle test (216 items per model), the base model scores 0.306 against a chance rate of 0.25. That is at 1B parameters and 10B training tokens, and the same-size transformer is also near chance. Instruction tuning did not change this: 0.301, p = 0.92. Recall works only near the end of the context and is at or below chance for the start and middle. Long-range recall is not solved. One structured recall test is not reported here, because we do not consider it a fair comparison between the two models.

State per conversation

We fed the model inputs from 952 to 31,926 tokens and read the conversation state after each one. The state was the same size, to the byte, at every point. Measured on one consumer GPU with fp32 weights, which take 3,651 MiB on the card.

Measured in the live demo on 11 November 2026: a session of the transformer twin takes 256 MiB at its 1,024-token context limit, against about 5.2 MiB for a Z-Next session.

These measurements say nothing about speed. On one NVIDIA RTX 5070 Ti (16 GB), sustained generation is roughly 48 tokens per second, single-stream and unoptimised; each step is slower than in a comparable transformer, and batched serving across hundreds of live sessions has not been measured. Earlier material compared session counts. Those were memory admission calculations for one configuration and measured no throughput, so this card leaves them out.

Limitations

• Long-range recall is near chance. See the needle test above. Through the demo pipeline, one chat build found a planted fact in a document of about 500 tokens and missed it at all three longer lengths tried, from about 2,700 to about 18,000 tokens. The informal demo tests of 11 Nov 2026 went better for a short code and no better for other facts (see What you can check yourself).
• Multi-step reasoning is essentially absent. None of the two-step chained questions in our test was answered correctly, by the base model or by two tuned checkpoints.
• It makes things up, often. That is expected at 1 billion parameters and 10 billion tokens. Examples we observed: the Moon's orbital period given as 21 days, and "the first three primes are 1, 2, 3".
• Chat quality is not product grade. The product variant failed 3 of its 17 acceptance gates.
• Format following is weak. On one chat build, a required answer format was not produced in any of 25 runs, although the correct answer appeared in the surrounding text in 80% of them.
• Some tasks trail the baselines. LAMBADA accuracy is behind both Pythia models (30.5 against 36.6 and 42.2), and WinoGrande is behind the twin (51.6 against 53.2). MMLU is at chance for every 1B-class model in the table, so it does not separate them.
• Speed. Each step is slower than in a comparable transformer. The single-stream figure is under Evaluation.
• English only. Every evaluation on this page is in English.

Not done yet

• Training beyond the 1B scale. Z-Next has not been trained at any larger size, so behaviour beyond 1B is unknown.
• A matched comparison with other fixed-state architectures at the same budget. Mamba-2, RWKV-7 and Gated DeltaNet also keep a fixed state; we have not compared against them.
• A twin comparison with an identical pre-training recipe.
• Independent reproduction by another party.
• Repeated runs with different seeds. Every result here comes from one run.
• Throughput measurement across many live sessions.
• Any safety tuning (see below).

Risks and safety

No safety work has been done on this model. There is no RLHF or other preference tuning, no red-teaming, no content filter and no bias evaluation. Biases in FineWeb-Edu and English Wikipedia pass through unmitigated.

• Output can be false, offensive or biased, and it can sound confident while wrong.
• It has had no training to refuse harmful requests.
• Do not use it for advice or for any decision about people.
• The weights are not public, which limits who can run it. Access is through an evaluation run with us, under NDA.

Protections in the deployed systems, and the ones that are missing, are on the system card.

Environmental impact

We report GPU-hours because we measured them. We did not meter energy, and we do not estimate emissions from an assumed power draw and grid mix.

How to access

• What we offer. We do not sell the 1B model as an off-the-shelf licence. We build a model at the size your product needs and license that to you: a custom Z-Next, or a conventional open language model trained or fine-tuned for you (instruction tuning, tool-use and agent tuning, evaluation). It can run on your own GPU. See what you can order.
• Hugging Face. tetracta/tetracta-znext-1b is a documentation and benchmark card. It holds no weights. Licence: tetracta-evaluation-only.
• Weights. The weights are not published. Under NDA we show enough of the architecture's properties for a technical evaluation.
• Joint evaluation under NDA. We run Z-Next on your workload, measure state size and behaviour on your hardware, and share the results with you. Under agreement we also share the evaluation harness details, raw per-item benchmark outputs, training telemetry and the reproduction protocol for every table on this page. We do not share the mechanism, the model shape or the inference implementation.
• Public demo. Open the live demo. The demo is switched on by hand, so it may be off when you visit. If it is off, ask us to switch it on.
• The control model. VAN-1B has public weights, so the baseline in every twin comparison can be checked. See its model card.

Citation

@misc{tetracta2026znext1b,
  author = {Tetracta},
  title = {Z-Next 1B: a language model with constant per-conversation state},
  year = {2026},
  howpublished = {\url{https://www.tetracta.ai/models/z-next-1b.html}}
}

Changelog