LLM training pipeline

The stages

The pipeline produces both standard transformers and our own architectures. A model passes through six stages, in this order:

1. Data collection and cleaning. Our own models were trained on FineWeb-Edu and English Wikipedia. Cleaning includes filters for adult content and spam.
2. Mixing, corpus build and tokenizer. Sources are mixed into one training corpus, and we train a SentencePiece tokenizer with a 64k vocabulary.
3. Pre-training. The engine has two branches: a standard transformer, and one with Rational Attention, our drop-in replacement for softmax attention. It trains across GPUs with DDP or FSDP2 and a distributed Muon optimiser. Before a long run starts, a fit check confirms the configuration fits the GPUs it is given.
4. Quantisation. To int8 and int4. On a 3B Rational Attention diagnostic checkpoint, int8 changed held-out loss by 0.0000 bits per byte and cut the checkpoint from 14.08 GB to 3.72 GB; int4 changed it by +0.0005 bits per byte. The saving is in memory only.
5. Instruction tuning and evaluation. An SFT pipeline, tool use, a blind judge panel that compares answers without knowing which model wrote them, and classic benchmarks.
6. Release and operation. A controlled release path to Hugging Face, operation on rented cloud GPUs, and a desktop GUI to run the pipeline.

What it has produced

VAN-1B and Z-Next 1B share the same corpus and tokenizer. VAN-1B's validation perplexity is 13.56 (3.7610 bits per token), measured with an older metric. On 10 zero-shot benchmarks, Z-Next 1B scores 45.24 and VAN-1B 44.62, a difference of +0.62 with an interval of −0.06 to +1.28, so the two are statistically level.

The two training recipes were not identical: a small share of Z-Next's pre-training windows also carried program-generated material that the transformer did not see. We read the result as no quality cost for Z-Next's design. It does not show an architectural advantage.

The Rational Attention runs are provisional and diagnostic in scale. Their results, with data downloads and checksums, are on the research page.

Multi-GPU training

The Rational Attention branch runs at MFU parity with a tuned softmax baseline, for at most 1% extra compute.

With FSDP2 and our distributed Muon optimiser, we have also trained a 3B model and a 7B mixture-of-experts model on two H200 GPUs, in short runs. Our note on Muon under FSDP describes that work and what it did not measure.

What is not done yet

• Large models. We have trained three language models from scratch: our own 1B model and its transformer twin on about 10B tokens each, and a third 1B model on a 60-billion-token corpus, which we keep private. Diagnostic runs go up to 7B. Larger configurations have only been checked for fit.
• Compute budgets. The 3B and 7B runs were short diagnostic runs of at most 0.236B tokens, well below a compute-optimal budget for models of that size.
• Multi-node. Training efficiency across several machines has not been measured.
• Optimiser economics. We have not measured whether Muon is cheaper than AdamW end to end on our data, so we do not claim it is.
• Very large models. Past roughly 80B parameters, FSDP alone is not enough. Tensor and pipeline parallelism are future work.
• Safety tuning. RLHF and red-teaming are not stages of the pipeline today.
• Training data. The models listed here were trained on English public data. A client's model is trained on the client's own data.

What you can order

We size the model to the job and train it for your problem and process. You can order any of these alone:

• A model trained from scratch. A custom Z-Next, or a conventional transformer, built at the size your product needs. See how access works.
• Fine-tuning of an open model. We take an open language model and fine-tune it on your data, so that it follows your instructions, calls your tools and works as an agent inside your process.
• Evaluation. Of the model we trained, of one you already run, or of a vendor's: full benchmark sets, a paired confidence interval for every difference, a blind judge panel, and pass or fail gates written before the run. Our own product variant passed 14 of 17 gates and is recorded as a fail. We can also measure what each of your languages costs in tokens across candidate tokenizers, as in our 204-language Token Tax study.
• Deployment on your own GPU. The model runs on hardware you control, with no public API, so your data does not leave. If it has to fit a smaller card we quantise it to int8 or int4.

The smallest job is fine-tuning an open model on your own examples and running it on one GPU you already have.

The steps of a training job

1. Data. We clean and filter your corpus and build a tokenizer if you need one.
2. Size and hardware. We choose a configuration and check that it fits your GPUs, or rented ones, before the long run starts.
3. Measurements first. We agree with you, before training, what will be measured and what counts as a pass.
4. Training. Pre-training from scratch, or fine-tuning of an existing model, across several GPUs where needed.
5. Compression. Quantisation to int8 or int4 if you need a smaller model.
6. Tuning and evaluation. Instruction tuning, blind judging and classic benchmarks, reported against the measurements agreed in step 3.
7. Release. The weights and the evaluation results, delivered where you need them.

Assistants grounded in your documents and tools

A small model is a poor store of facts, so we build assistants that send each question to the right tool or document, compute numbers in code and cite their sources. They can run on your premises. The assistants we have built this way run on our own 1B models. The results are qualitative (see the routing note in our research), and those models are English only.

Data only

If you train elsewhere, we run only the first stages: collect and clean your text, filter adult content and spam, remove near-duplicates, mix the sources and tokenize the result. You get a training-ready corpus and a report of what was removed and why. This is the path that prepared the corpus behind VAN-1B and Z-Next 1B.

Training-run stability audit

Send us the gradient-norm log of a run, or let us log a new one. We put it on a process-control chart and report excursions and a capability index (Cpk), the same measurement behind our Rational Attention results, so "stable" becomes a number you can compare across runs. It measures stability and says nothing about model quality.

When a custom model makes sense, and when it does not

Often the better choice is to fine-tune an open model. When it is, we say so, and we can do that work. It is usually the case when:

• an open model already handles your task and only has to learn your instructions, your tools or your output format;
• your sessions are short, so the state a conversation holds never becomes the limit;
• you want to stay on a model that other teams and tools already support.

A model built for the product makes sense when the product sets a condition the model has to be designed around:

• each session must keep a fixed-size state on your own hardware, however long it runs;
• many long-running sessions have to share the same GPU;
• the product must run offline, or on a device whose hardware is known in advance;
• the data cannot leave the building.

Our own Z-Next 1B is an example of the first condition: its state stays at about 5.2 MiB per conversation, and the model card has the measurements. An open model that we fine-tune also runs on your own GPU, so the last two conditions call for a custom model only when no open model fits the device. The sizing review described below tells you which case yours is.

How a project starts

You send a set of example inputs with the answers you want, and tell us which GPU you plan to run on. We write back within one working day to say whether the job is one we can take. If the task is already known, the first job can be the fine-tune itself. Otherwise it is a sizing review that lasts one to three weeks, in which we evaluate one or two open models on your examples. Either way the scope and price are fixed in writing before the job begins, and you owe nothing for it until you accept what it delivers. Any larger training run is quoted only after that first job, at the size it showed the work needs.

What you receive

• From a sizing review: a written report on how the open models did on your examples.
• In that report, whether fine-tuning is enough or the job needs a model trained from scratch, and at what size.
• From a fine-tune: the tuned weights, running on your GPU.
• With the weights, the evaluation results against the measurements agreed before training.

If you do not have example inputs and answers yet, describe the task and we will say which examples to collect.