★ SFT anatomy · 2 models on HF · 30 Jul 2026
We trained a 0.94B transformer from scratch on 10.32B tokens, one epoch, instruction-tuned it, and measured what SFT actually changed: 0.68% of the weights (5.35% in the embedding, 0.04% in the norms), output entropy 5.95 → 1.80, and an 11.5% pre-training loss penalty paid almost entirely in the first round. Then we ran the byte-identical SFT on a second model with a different architecture and bit-identical data: post-SFT peak probability 0.730 vs 0.722, embedding shift 5.347% vs 5.327% — and in the non-transformer, where memory is separable, SFT moved the memory by one part in a million. Honest parts: we reach parity with a same-size base model on ~1,750× less data but lose clearly to instruction-tuned rivals; the alignment tax tripled on the second architecture against our own pre-registered prediction; and one diverging eval sequence shifted a mean bpc by 0.46. Weights + tool runtime public on Hugging Face.
Read the study →
★ Model merging · model on HF · 22 Jul 2026
We stitched two specialists into ONE 3B model with a training-free TIES merge: +13 and +26 points on the target skills, zero measured general regression (Belebele-EN, perplexity and chat identical to base). Across ~10 experiments we validated the law synergy = headroom × complementarity × proximity — including a clean negative control that turns a merged coder into mush. And the honest part: three attempts at internally-guided merge surgery did not beat blind TIES; what internal measurement reliably does is predict which pairs merge profitably, before merging. Model public on Hugging Face.
Read the study →
★ Language economy · timely · 21 Jul 2026
Everyone is benchmarking Kimi K2 and hyping K3. We added Kimi K2's tokenizer to our 204-language leaderboard and asked the one thing you can verify yourself: it lands 4th of 9. The genuinely interesting part is narrow — Kimi carries a 28% larger vocabulary than the DeepSeek-V3 architecture it is built on, yet is marginally less efficient across 204 languages, spending its extra tokens mostly on Chinese (where it is the single best we measured). A trade, not a rout — and strictly efficiency, not quality. We measure; we don't hype.
Read the measurement →
★ Language economy · pre-registered · 21 Jul 2026
The token bill for a non-English answer has two parts, and we separated them across two model families and seven sizes. The tokenizer tax (tokens-per-character) is size-independent — a clean cross-check of our leaderboard from the output side. The generation side is behavioral: every model answers in non-English at ~0.3–0.7× its English length. Pre-registered, reproducible, with the honest limits stated (it is a length signal, not a quality verdict; the size-trend is mild and noisy).
Read the study →
★ Language economy · 204 languages · 20 Jul 2026
The same meaning costs a wildly different number of tokens depending on the model — and tokens are what you pay for (API price, GPU time, real context size). We measured 9 open tokenizers (now including Kimi K2) across 204 languages on a standard parallel corpus (FLORES-200). Off English the token tax runs 2–3× and up to 10×; big-vocabulary multilingual tokenizers (Gemma-2, BLOOM) win, 32k-vocab models are the most expensive. Reproducible, $0, tokenizer-only — and strictly an efficiency measurement, not a quality one.
See the leaderboard →
★ The autopsy · pre-registered · 19 Jul 2026
Fine-tuning concentrates, quantization smears — a finding clean enough to distrust. Six pre-registered assassination attempts across four model families: an instrument-floor control that killed our onset claim (twice), a public metric correction, a normalization challenge, the day our engine accused a correct label, and a fourth-family test (Gemma) that missed — published in full. The honest scoreboard: band-exceeding in two families, one inconclusive, one miss.
Read the autopsy →
Training methodology · 5 Jul 2026
Estimating a large model from small runs is only as good as the small measurements. Four traps that fooled us first, with the numbers: a per-size effect sitting ~40× below step-noise (~0.002 BPB), early leaders that flip below Chinchilla, a domain-sorted pipeline faking a 0.11 BPB edge (held-out PPL past 3000), and aggregate metrics hiding per-domain effects. Measurement hygiene, not a fitted law — limits stated.
Read the full note →
Training systems · 30 Jun 2026
We ran Muon under FSDP and proved the distributed update is bitwise-identical to single-GPU Muon (max|diff|=0). Measured on 2× H200: 3B and 7B train and learn at 16–20% MFU/GPU. What we measured vs what we only projected, stated plainly.
Read the full note →
★ Full data · measurement floors · 13 Jul 2026
Five newly trained 0.93B runs pin the seed floor; fp16/TF32/batching pin the numeric-jitter floor on a converged MoE; and against both, one finding survives: real int8 kernels flip ≈2× the routing decisions of the RTN simulation most quant evaluations use (1.47M decisions, deterministic). Plus a ρ=0.95 correlation we decomposed and refused to publish. Tables, thresholds pre-registered.
Read the full note →
★ Results · internal diagnostics · 10 Jul 2026
Two 0.93B models, identical but for the attention operator, looked different inside our own scan. Then a pre-registered seed-null control on three newly trained models showed every internal difference was within the noise between two ordinary runs — so the figures were corrected and the full control data published. A worked example of negative-control measurement; the loss-twin result and the custom-architecture scan capability stand.
Read the full note →
Training methodology · 15 Jun 2026
A Six-Sigma view of training stability: treat the gradient-norm as a process, put it on an SPC control chart, and turn "stable" into a measured, auditable number — even something you can put in an SLA.
Read the full note →
Quantitative methodology · 15 Jun 2026
With limited data the shortest forecast horizon overfits first — and looks exactly like a real edge until it meets reality. The temporal placebo, leak-free point-in-time backtesting, and out-of-sample discipline that tell them apart.
Read the full note →
Small-model engineering · 15 Jun 2026
A small model is a bad encyclopedia — and that's fine. The useful design makes it a fast, cheap orchestrator that routes to tools and grounds answers in sources, instead of memorizing facts in its weights. Honest results, honest limits.
Read the full note →
Applied side-project · predictive maintenance · 21 Jun 2026
Zero-training, sensor-agnostic condition monitoring, with measured results on real run-to-failure test datasets: bearing-degradation monotonicity +0.96/+0.98 (Wilcoxon p<0.001), alarms before failure, and multi-modality coverage (vibration, sound for pumps, thermal). Honest results, honest limits — plus a live public demo and an integration guide. Not yet field-deployed; pilot partners welcome.
See results & capability →
More notes as the work continues. For the technical brief or the reproducibility evidence chain, get in touch.
