Research Note · full data

Noise Floors for Internal LLM Measurements: Seed, Numeric Jitter, and Real-vs-Simulated int8

Tetracta AI Teams · 13 July 2026

This is the data companion to the twin-study note: the measurement tables from a three-day control campaign (~$300 of compute, five newly trained 0.93B checkpoints, one converged public MoE, every threshold sealed before the corresponding run). Tables, not stories. No method internals are disclosed — metrics are aggregate outputs of the scan.

1 · Seed floor: five 0.93B runs, one recipe

Same trainer, same 10.3B-token corpus in the same order, same schedule; only the init seed varies (and the attention operator, between the VAN/softmax and RAT/rational arms). All at step 30k of a 157k plan.

int8 = per-row round-to-nearest on all linear weights (routers excluded); disturbance = mean absolute internal deviation between each model's own fp32 and int8 fingerprints, dense 378-probe grid.

What the table says. Loss: the spread between softmax seeds (0.0020 bpb) exceeds the softmax-vs-rational gap (0.0010) — the operator moves loss by less than the seed does. Internal metrics: within-operator seed spreads are 7.3× (disturbance) and 3.2× (flip rate) — larger than any between-operator difference we had reported, which is why those figures were corrected. Cross-scan divergence tells the same story: operator pair 0.000307 vs seed pair 0.000324 (ratio 1.06), both spanning 84% of depth.

2 · Numeric-jitter floor, and real-vs-simulated int8

On a converged public MoE (Qwen1.5-MoE-A2.7B, 60 documents, 61k tokens, 1.47M routing decisions), we measured how much everyday numeric variation moves token-level routing — the denominator any quantization claim must beat — then the quantization arms against the same baseline (bf16).

Router/gate weights stay full-precision in every arm (verified bit-identical to baseline at load — the industry-standard protection, structurally enforced). Determinism checks: bf16 double-pass 0.0000%, int8 double-pass 0.0000%, int8 reload 0.0000% — the flips are stable properties, not run noise.

Three readings. (i) Everyday numeric changes nobody audits (fp16 casting, batching) already flip ~0.85% of top-1 routing decisions — any quant number must be read against that floor. (ii) The real int8 kernel flips ≈2× more routing decisions than the round-to-nearest simulation most quant evaluations use (2.60% vs 1.30%) — and the dequantized-weights arm pins the mechanism: weight error alone reproduces the simulation, so the extra factor comes from the int8 compute path. Anyone auditing quantization with a simulation is seeing about half of the real kernel's effect on routing. (iii) Perspective: top-1 flips almost never leave the old top-4 set (0.011%), and flipped tokens sit at near-tie router margins (99.9% below the median margin) — so we do not claim this "damages" the model; that would require a downstream-harm link we tested for and did not find (§3).

3 · The correlation we refused to publish

Nine quantization configurations (int8/int4/nf4 × granularity × depth-restricted), each measured for routing-flip rate and for greedy-behaviour change against baseline:

The raw Spearman correlation between flip rate and behaviour change is ρ = 0.95 — comfortably past the GO threshold we had sealed (≥0.6). We are not publishing it as a finding, for three reasons the table itself contains: both variables ride the same axis (quantization aggressiveness), so the correlation is largely a tautology; controlling for Δppl collapses it to a partial ρ of 0.58, which at n=9 is not significant (p≈0.13); and the iso-ppl rows are decisive — deep-half (0.42% flips), shallow-half (1.23%) and g128 (1.14%) differ 3× in flip rate at essentially zero Δppl, yet produce identical behaviour change (43.3% each). Flip rate carries no predictive information beyond perplexity here. A correlation you do not decompose is a press release, not a finding.

4 · What these floors are for

Every number in our diagnostics now ships against three denominators: the seed floor (§1 — how much two ordinary runs differ inside), the jitter floor (§2 — how much everyday numerics move the same metric), and a determinism check (is the effect stable at all?). That is the discipline the three-day campaign bought, and the reason its main product was two corrections and one surviving finding rather than four headlines. The surviving finding — real kernels vs simulation — is the one we would want a reader to take away: if your quantization evaluation uses a weight-rounding simulation, you are not measuring your deployment.

Artifacts. The five 0.93B checkpoints and scan summaries are on Hugging Face (sha256-sealed). Metrics here are aggregate scan outputs; the instrument's calibration and internals stay closed. Companion narrative: the twin-study note.

— Tetracta AI Teams · for humans, like humans.

← All research notes Talk methodology