The Muon Optimizer Under FSDP, Verified Bit-for-Bit
Tetracta AI Teams · 30 June 2026
A results note. Every number below was measured on real hardware (2× H200), with real data and our production trainer, and re-checked against the code. We say plainly what we measured and what we did not.
The Muon optimizer has become the most interesting thing in training efficiency this year, and the most common reason teams give for not using it at scale is mechanical: Muon's Newton–Schulz step wants the full two-dimensional weight matrix, and FSDP — the standard way to fit a large model across GPUs — shards exactly that matrix across ranks. The two assumptions collide. So the usual move is to avoid the combination, or to reach for a heavier parallelism stack. We wanted Muon and FSDP, on the hardware we actually have, so we built the bridge and then did the unglamorous part: we proved it was correct before we trusted it.
Distributed, and bitwise-identical.
Our distributed Muon gathers the full gradient for each matrix, runs the identical Newton–Schulz iteration on the full matrix, and scatters the result back to each rank's shard. Because the gathered matrix is the same on every rank, the math is not an approximation of single-GPU Muon — it is single-GPU Muon, computed in a sharded world. We did not want to take that on faith, so we wrote an equivalence gate: run the distributed optimizer in a genuinely sharded setup, run plain single-GPU Muon on the same inputs, and compare the updates. The result, across four matrix shapes, is max|diff| = 0.00e+00 — bitwise-identical, not "close." It holds across multiple steps with momentum and Nesterov active, with weight decay, in bf16, and with uneven shards. A correctness claim you can re-run is worth more than one you assert, so we built it to be re-run.
It trains, and it learns.
On 2× H200 we trained both a 3B and a 7B model (mixture-of-experts) under FSDP with the distributed optimizer. They train without crashing — the 3B ran 60 steps plus evaluation cleanly — and they learn: the training loss, expressed as bits-per-byte, falls steadily (3.3 → 2.1 on the 3B). Model-flops-utilization lands in the 16–20% per GPU band depending on batch size. Two numbers we found worth keeping: the larger model is the more efficient one at the same batch (7B at 19.7% beats 3B at 16.4%), because the fixed costs amortize better as the model grows; and FSDP's overhead versus a single-GPU run is about three points of MFU — a modest, measured tax in exchange for 2× throughput and the headroom to go past what one card can hold.
A real bug, found and fixed.
The path was not free of teeth. We hit a deterministic distributed hang — the kind that reproduces even on a quiet host, which is how you know it is your code and not the cluster. With sparse mixture-of-experts routing, some experts receive no tokens on a given rank; their weights then never enter that rank's computation, their gradients come back empty, and FSDP's gradient synchronization becomes asymmetric across ranks — which is exactly the condition that deadlocks the collective. We fixed it at the model level in a way that is behavior-neutral: the numerics are unchanged, the equivalence gate still passes bitwise. Sharding the optimizer's momentum buffer, in turn, is what let the 7B fit at all. None of this is novel physics; it is the ordinary, careful debugging that stands between "it ran once" and "we trust it."
What we measured, and what we did not.
We would rather a reader find the limits stated here than discover them later.
The engine is built to scale — presets up to roughly 80B, a launcher that takes the GPU count you give it, and a cost-aware estimator with a smoke fit-check so a long run is not started on too few cards. But built is not the same as run. We have only trained 3B and 7B, on two GPUs. The 14B–80B figures are projected and smoke-checked for fit, not trained, and we will say so until the day we have the hardware to run them. Past roughly 80B, pure FSDP runs out of road and you need tensor- and pipeline-parallelism; that is future work, not a current capability.
Two more honest edges. This is not a first — Moonshot shipped a distributed Muon before us; what we add is a build whose correctness is verified bit-for-bit and integrated into a real trainer. And the question everyone actually cares about — whether Muon is genuinely cheaper end-to-end on our data, once its token-efficiency is weighed against the throughput it costs — we have not measured yet. We know exactly which single experiment settles it, and until we have run it we will not call this faster or cheaper. It is, for now, a correctness-and-systems result, and we are content to let it be only that.
Why we wrote this down.
Most of what gets shared in this corner of the field is a headline number with the methodology left as an exercise. We find the opposite more useful: a result you can interrogate, with the measured and the merely-projected drawn in different ink. That habit is the whole point of how we work — we pressure-test our own claims first, and we publish the weak cases alongside the strong ones. The bitwise-equal gate is the kind of evidence we trust; the unmeasured economics is the kind of gap we name out loud rather than paper over.
If you are training with Muon, fighting FSDP, or just curious about any of the numbers above, we are happy to compare notes.
— Tetracta AI Teams · for humans, like humans.