When the Small Run Lies About the Big One
Tetracta AI Teams · 5 July 2026
A methodology note. Every number below came off our own training runs on real hardware and real data. We share what we measured and what we did not; the mechanism, formula and recipe are never disclosed.
Training small models to estimate a larger one is one of the most useful moves in the field — and one of the easiest to get quietly wrong. The published debate is usually about which functional form fits the size–loss curve best. In our own experience the functional form was rarely the thing that bit us. What bit us was the measurement underneath the curve: the points we were fitting were noisier, more fragile, and more scale-dependent than they looked. A scaling law is only as honest as the small measurements it's built from, so this note is about four traps that fooled us first, with the numbers that gave them away. Our scales are modest — models from 601M to 7B parameters, single seed, deliberately below Chinchilla — which is exactly the regime where these traps live.
1. The signal can sit far below the noise.
On a 1B model trained to roughly 10B tokens, we compared a drop-in variant against its baseline as a paired run — identical seed, identical data order — so the two curves could be differenced step-for-step. At a clean held-out checkpoint the gap was about 0.002 bits-per-byte (the baseline microscopically ahead: 1.0397 vs 1.0418, perplexity 27.8 vs 28.0), and window-averaged differences stayed in the 0.001–0.0014 range without growing. That gap sits roughly 40× below the step-to-step noise of either run.
The trap: an earlier, unaligned glance had shown an apparent ~0.06 BPB gap that looked decisive — roughly thirty times the real difference. It was pure step-misalignment: a low-step point of one run read against a high-step point of the other. At the same step, differenced properly, the real gap was the ~0.002 above. If the per-scale effect you intend to extrapolate is far below your run-to-run noise, then unpaired points don't measure it — and a curve fit through noisy points inherits the noise, whatever its functional form. This is the part that speaks directly to the "which scaling law" question: when the residuals of your fit are dominated by measurement variance rather than by scale, you are selecting the functional form on noise, not on the model. Tighten the measurement first; choose the form second.
2. The tuning you inherit is wrong twice over.
Our small diagnostics were data-limited on purpose — one was roughly 85× below Chinchilla-optimal (about 238M tokens). Two things reliably go wrong in that regime. The optimal learning rate is inflated relative to a well-fed run, so a setting tuned small is the wrong setting large. And partial, not-yet-converged runs show an early leader that is not the converged winner — we watched the ranking between two setups change as the runs finished. Extrapolating from an undertrained proxy can therefore mis-rank the very thing you're trying to predict. The discipline that helps is dull but decisive: converge far enough (or toward a Chinchilla-optimal token budget) before you trust an ordering, and re-tune at scale rather than carrying the small-scale hyperparameters forward.
3. Pipeline artifacts masquerade as scaling signal.
Early on, a data pipeline that fed domains in order — code first, prose later, with no global shuffle — handed us a phantom 0.11 BPB "reversal," a gap large enough to look like a real result. It wasn't. It was a train-fit / out-of-distribution artifact, and the tell was the held-out perplexity, which had blown past 3000 — a number that says "your evaluation is off-distribution," not "your model is better." A global shuffle and a matched held-out set, and the edge evaporated. A surprising amount of what reads as scaling signal is really pipeline hygiene; before you fit a curve, confirm every point was measured on the same, un-corrupted footing.
4. Aggregate metrics hide the effect.
The last trap is structural rather than numerical, and the quietest for it. An aggregate held-out loss can average a domain-local effect away to nothing: the difference that should move a decision only surfaces once the evaluation is split by domain — code, math, science, general prose — and stays invisible in the mean. We keep every held-out eval split for exactly this reason. Measurement granularity is itself part of the measurement, which is why we extrapolate the protocol — the same eval slices, the same metric definition, the same footing — as carefully as we extrapolate the loss.
What this is, and what it is not.
We would rather state the limits than have a reader find them. These are measurement-hygiene lessons, not a fitted scaling law: single seed, sub-Chinchilla diagnostics, models in the 601M–7B range. We do not publish a fitted exponent, because we have not run the controlled, multi-seed scaling sweep that would earn one, and we would rather say that than imply a rigor we haven't spent. What we do have is a set of failure modes that cost us real time — the noise floor, the flip, the pipeline artifact, the hidden per-domain effect — offered here in case they save someone else the same week.
Why we wrote this down.
Most of what circulates in this corner of the field is a headline number with the methodology left as an exercise. We find the opposite more useful: the results and their limits in the same paragraph. That is the whole of how we work — we pressure-test our own claims first, our own results included, and we publish the weak cases next to the strong ones. If you're estimating large models from small ones and any of these numbers ring true, we're happy to compare notes.
— Tetracta AI Teams.