Eric Bigelow on why an LLM’s decision is a coin flip
A Goodfire interpretability researcher argues the moment an LLM “decides” is the stochastic sampling of a token, not the readable reasoning before it — which is why chain-of-thought monitoring is fragile and why he wants interfaces that surface the model’s uncertainty.
The decision is a coin flip at sampling
Bigelow’s core claim is that an LLM does not “decide” inside its reasoning — the deciding event is the random draw that samples the next token from its final distribution.
models even to some extent when they plan out their responses what they're actually going to say at every single moment is really determined by like the flip of a coin which is what token will it generate next
The answer snaps shut on a meaningless token
Resampling thirty rollouts at every token of a reasoning chain shows the answer distribution often holding steady, then collapsing abruptly — sometimes on a character as empty as an open parenthesis.
often the distribution just kind of collapses onto a final answer as the model sort of you might say decides what it's going to say next
In-context learning is everything the weights aren’t
He defines in-context learning broadly: not just few-shot prompts, but every way a model adapts its behavior that is not stored in its weights — including reasoning, where it learns from text it just wrote.
what I think of in context learning is capturing is everything the model is doing to adapt its behavior that's not happening with in weights learning
Context and memory beat fine-tuning each user
His short-term bet is a bitter lesson: context engineering, harnesses and memory will out-compete per-user fine-tuning, because a model’s memory is just tokens it re-reads and learns from in context.
the memory is just the way I see it, a bunch of data that is tokens that are in context learned
Reasoning is a misnomer — it’s a soup of tokens
Modern reasoning models don’t march A to B to C; they enumerate every possibility into a “soup of tokens,” then go back and skim one answer out — so the real selection happens at the very end.
I think reasoning is almost like a misnomer for what reasoning models are doing
The chain of thought is drifting out of English
Reinforcement learning that rewards only the final outcome gives the model no reason to stay human-readable, so chains of thought and sub-agent messages drift into private dialects — and pushing back makes them less faithful, not clearer.
there isn't necessarily an incentive to keep things human readable
The scariest behaviors need frontier scale
The behaviors worth studying for safety — like reward hacking — only emerge once a model passes a capability step function, which outside the big labs means leaning on frontier open-weight models.
just having the model pass that step function just in terms of some some particular behavior coding ability was very important to start also getting to see these really these dangerous behaviors that we want to study and understand better
Show the user how unsure the model was
Rather than wait for models that never guess, he wants interfaces that surface the model’s uncertainty — flagging when an answer stated as fact was really a near coin-flip that might be a hallucination.
the more that can be surfaced to the user and the more the user can know how uncertain was the model about this thing that it reported as a fact when it may have been a hallucination