Frontier LLMs edit weights directly — Weight-space programming — What does it mean?

The news. In early November 2026, a research team released the blog post "Sparks of Weight Space Programming" with the complete logs of every experiment. In one of them, OpenAI's gpt-6-astra, used with no gradient descent, edited Qwen3.5-0.8B and raised its reported MATH-500 score from 7.9% to 48.1%. The authors describe the results as early. Read the logs →

Picture the model as a mixing desk. A model's behavior lives in numbers, so you can change the behavior by changing the numbers directly, with no training loop. Every layer is a bank of channel strips, and each strip adds its own sound to one shared master mix, which is the residual stream. In the feed-forward network, the last matrix is the down-projection, and each of its columns is the vector one neuron adds to the mix when it fires. If a few strips keep pushing a stuck hum into the mix, the song replays the same bar forever. The usual repair, gradient-descent fine-tuning, is re-recording the album: collect data, run many passes, and let a loss decide how every strip should change.

Weight-space programming skips the studio. The editor, OpenAI's gpt-6-astra, was given a Python sandbox over the weights of Qwen3.5-0.8B, practice problems with answers, and a budget of 10, 25 or 50 forward passes of the model. Gradient descent, backward passes and any training were forbidden, and it never saw the test benchmarks. The failure it found was not bad math; it was thinking that never stopped. In its first 96 practice problems, 85 ran to the 32,768-token limit with the model repeating itself. Comparing contexts where the model was making progress against contexts where it was looping, the editor located a repetition direction u, then found the MLP columns that write most along it. The final edit removes that hum from the 64 channels in layers 15 and 19 that write it most. Other layers, the embeddings, the output head and the final norm were left alone.

The 10-pass run shows the arithmetic of such an edit in its plainest form: W' = W - 1.5·u(uᵀW). For each column w, the term u(uᵀw) is the part of w that points along u. Subtracting 1.5 times that part turns a component of size c into c - 1.5c = -0.5c, so the hum is reversed and halved, not just removed (this assumes u has length 1). The 50-pass run, which produced the big gain, describes its edit as subtracting the loop direction from those 64 columns per layer; the logs include the exact diff against the public weights (README). Because only the component along u changes, a neuron that mostly writes something else keeps doing so, and the editor also ran dose-response, layer-isolation and random-direction controls before freezing the edit.

Here are the numbers that matter, held to one setup. The 50-pass edit touched 2 layers × 64 columns × 1,024 values = 131,072 weights, about 0.016% of the model, and MATH-500 answers cut at the token limit fell from 93.0% to 33.1%. On 1,152 held-out practice problems (not the benchmark), the paired comparison went from 10.6% to 60.4% accuracy, with 579 problems rescued and 5 broken, a net 574 of 1,152, or +49.8 points. 576 of the 579 rescued problems were answers that had been cut off at the token limit, and accuracy on answers that already finished did not change. So the edit did not teach the model mathematics; it let a model that already had some ability finish its thought. The editor itself writes that it expects the 10-pass edit "to help by reducing repetition, not by adding mathematical knowledge".

Three limits keep this honest. The result is one run per forward-pass budget, so 48.1% shows what is possible, not what to expect on average; the 10-pass and 25-pass runs reached only 12.3% and 14.1%. The target was a small model whose base failure rate was unusually high (93.0% of answers cut off), so fixing the looping could recover a lot; a model that rarely loops, such as the Gemma 3 270M used in the authors' follow-up (8.5% of its answers reach the limit), would have less of this slack to recover. And the conditions are specific: thinking mode on, temperature 0.6, a 32,768-token cap, and the authors call the results early. For a practitioner the durable lesson is not the headline score. It is that a behavior can sometimes be traced to a direction and removed by editing a few weight columns, which is cheaper than retraining when you can name the failure.

Goes deeper in: LLM Internals → Transformer Block → The Feed-Forward Network