OPEN-1B publishes a training run you can replay bit for bit — Step-replay auditing — What does it mean?
The news. On September 15, 2026, an arXiv paper (2609.17380) introduced OPEN-1B, a 1-billion-parameter training run built to be audited rather than merely downloaded. Alongside the model the authors release the full pretraining dataset, every intermediate checkpoint, the training code and an audit harness — and, crucially, they impose a deterministic order on GPU reductions, data batches and collective communication, so any published step can be replayed and compared bit for bit on different hardware. The paper describes this as a new tier of model transparency, fully auditable. Read the paper →
Picture the blanket again. You have the finished object, and you have a photograph taken after every single row. What you do not have is anybody willing to re-knit the whole thing to prove the pattern was followed. The move is to make one row checkable on its own: take the photo from the end of the row before, work the next row exactly as the pattern says, and hold your result against the photo that follows it. Match, and that row is settled — without touching any other row.
This only works because the pattern fixes the order. Work the same stitches in a different sequence and the row comes out recognisably similar and measurably different, and a mismatch then tells you nothing at all: you cannot separate "the pattern was not followed" from "I worked it my own way". A training step is order-sensitive in the same way, though for a different reason — floating-point addition is not associative. Adding a list of numbers left to right and adding the same list in pairs can give answers that differ in the final bits, because every addition rounds. So when thousands of GPU threads fold their partial sums into one number, the shape of the tree they fold in is part of the answer rather than an implementation detail. Change it — different hardware, a different batch split, a different library version — and the low bits can move.
OPEN-1B's answer is to pin down every place that order is free to wander. The paper names three: the order GPU kernels reduce in, the order the data-parallel cluster takes its batches in, and the order of the collectives that exchange gradients between and inside nodes. Pin the reduction order, and the numeric format you chose rounds the same way everywhere. Pin the batch order, and every replica — each GPU's own copy of the model — sees the same slice at the same step. Pin the collectives — the all-reduces that run inside a node and across the network between nodes — and the summed gradient stops depending on the order the messages happened to arrive in.
With those three fixed, a step turns into a plain function: checkpoint and data slice in, next checkpoint out, the same bits every time, even on hardware the original run never touched. That last clause is the load-bearing one. Bitwise determinism on one machine is an old trick and a weak guarantee, because an auditor who reproduces your run on your cluster has mostly proved that they have your cluster. Bitwise determinism across ordinary, mismatched hardware is what lets a stranger check you.
| Tier | What is released | What an outsider can actually check |
|---|---|---|
| Open weights | the model file | that it runs, and how it behaves |
| Open weights, data and recipe | the dataset and the training code | that the recipe is plausible — not that it produced these weights |
| Plus every intermediate checkpoint | the state after every step | where in the run the behaviour changed |
| Plus fixed operation order and an audit harness | a step that replays bitwise on other hardware | that a named step is exactly what was claimed, bit for bit |
Determinism on its own still leaves the bill unpaid. Replaying a full pretraining run on one machine is impractical — that is the paper's own framing, and it is what makes "just reproduce it" an impractical answer to "is this model what you say it is". So OPEN-1B splits the bill instead. Because every step now stands alone, audits distribute: one verifier replays step 12, another replays a step far later in the run, neither has seen the other's machine, and their separate results compose into coverage of the trajectory. The paper calls this a collective verification scheme, and it is the part that changes what open can mean — transparency stops being a pile of artifacts you are invited to trust and becomes a claim that anybody can spot-check.
Where the audit bill actually goes. Hold three things fixed: one released run, one step costing about an hour on a single commodity machine, and each volunteer willing to do ten steps (illustrative — the paper publishes the mechanism, not a step count or a replay cost). Say the run is 100,000 steps. Re-running it end to end is the option the paper rules out as impractical on one machine. Under step replay the arithmetic inverts. A hundred volunteers taking ten different steps each cover 1,000 distinct steps — 1% of the trajectory for ten hours of any one person's time. Ten thousand volunteers cover all 100,000. The total work has not shrunk by a single step; what changed is who carries it. Coverage grows with the number of auditors while the cost to any single auditor stays flat at ten hours — and a bill that never lands on one party is the kind an outsider can realistically pay.
Goes deeper in: GPU & CUDA → Tiling & Matmul → Tree Reduction
Related explainers
- MarginGate — Margin-gated verification for batch-invariant decoding — the same non-associativity, met at decode time instead of during training.
- Prefix caching made 36.2% of identical agent runs diverge — what happens downstream when nobody records the state a result depended on.