Latent Space

Danielle Perszyk on why reliable agents must model your mind

Danielle Perszyk· Researcher at Amazon AGI Lab at Amazon AGI Lab
·~49 min·English·Latent Space
AgentsReasoningAI Company
TL;DR

A cognitive scientist at Amazon's AGI Lab argues intelligence is fundamentally social, so the path to reliable agents isn't better button-clicking but AI that models your mind, aligns its representations with yours, and widens rather than narrows human thought.

01Core Mental Model

Intelligence Lives Between Minds, Not Inside Them

Perszyk's starting axiom is that intelligence isn't a property of an individual mind but something that emerges from the interactions between many, so AI built only for the engineers who make it misses where intelligence actually lives.

The big idea is that human intelligence is collective. Anthropologists say that we've got the the collective brain. No one individual is capable of even surviving on their own. We depend upon the collective.

— Danielle Perszyk, Latent Space
Key Insight
If intelligence is a property of the network rather than the node, the industry-wide race to build one maximally-capable model may be optimizing the wrong unit. The returns could live in the connections between many models and many people, not inside any single brain.
02The Diagnosis

Stuck in the Chatbot Attractor

The field has settled into a shallow comfort zone of chatbots, coding agents, and turn-taking in batches that Perszyk calls a local attractor, and it looks nothing like the real-time way humans actually think together.

We're we're kind of trapped in this local attractor state of chatbots and coding agents and like turn-taking in batches. And this is absolutely not how humans interact with each other.

— Danielle Perszyk, Latent Space
Key Insight
"Local attractor" is a precise choice of words: a system can be locally optimal and globally stuck. The danger isn't that chatbots are bad — it's that their fast, verifiable feedback loop makes the shallow basin feel like progress, which is exactly what keeps you from climbing out.
03Reframing Reliability

Reliability Is Modeling the User's Mind

Reliability was never about clicking the right button every time; it's about modeling the user's mind and decomposing their goals, preferences, and intentions as the task itself keeps changing under them.

So, ultimately reliability has less to do with clicking in the same place and scrolling and more to do with modeling the user's mind. And that shift is everything.

— Danielle Perszyk, Latent Space
Key Insight
This quietly redefines the product. If reliability means modeling intent, the moat isn't a better click-accuracy benchmark — it's the user model itself. The executive-assistant comparison implies the real bar is a relationship built over time, not another percentage point on a screen-control eval.
04Reframing Memory

Memory Isn't Storage, It's How You Simulate the Future

Treating memory as a place you offload to is a leftover from the 20th-century mind-as-computer metaphor; in humans memory is how you simulate the future, and agents need that kind of memory rather than a database bolted on the side.

memory is this thing that we kind of offload, but that's not at all how it works in humans. Memory is everything. It's how we simulate the future.

— Danielle Perszyk, Latent Space
Key Insight
Naming the metaphor is the move. "Mind as computer" made memory feel like a solved subsystem to attach later; calling memory "everything" implies you can't add it as a retrieval layer after the fact — it has to be built into how the model learns in the first place.
05The Training Objective

Stop Playing Whack-a-Mole With Tasks

Tuning a model to ace one task with reinforcement learning is whack-a-mole because the skill doesn't transfer; the objective that actually generalizes is the one humans run on, inferring other minds and aligning representations.

Humans are spontaneously constantly inferring the existence of other minds and we are optimizing for aligning them. We're optimizing for aligning our representations.

— Danielle Perszyk, Latent Space
Key Insight
Goodhart's law is the hidden warning: every task-specific benchmark becomes a worse measure the moment it's the target. Aligning representations is attractive precisely because it's an objective you can't reward-hack by memorizing one narrow environment.
06World Models

Every Human World Model Is a Social One

Human world models are social from the very first moment because you model how another mind reads the world, and that perspective-taking, not generative video, is what lets you adapt to an environment you've never seen.

but our world models from the very beginning are social world models. We are inferring how another mind is interpreting the world and we are inferring what their perspective might be

— Danielle Perszyk, Latent Space
Key Insight
This is a sharp break from the generative-video framing of world models. Perszyk's claim is that the unlock isn't rendering the world accurately — it's modeling the observer, and a system that only regenerates its own environment has almost no signal to learn from.
07The Risk

AI Is Quietly Narrowing How We Think

AI that regresses everyone toward the mean is quietly narrowing human thought, with individual scientists publishing more while the field as a whole contracts, and Perszyk's fix is a diverse society of AIs rather than one monolith.

individual scientists who are using AI tools are benefiting because they're producing more papers, they're getting more grants accepted, but science as a whole is narrowing. And that is terrifying.

— Danielle Perszyk, Latent Space
Key Insight
The unsettling part is that the harm is invisible in the moment — you accept each suggestion below your own threshold of awareness. It reframes AI safety as an epistemic-diversity problem: the failure mode isn't a rogue model, it's a fleet of helpful ones quietly converging everyone on the same answer.
08The Synthesis

Alignment Is the Solution, Not the Problem

The alignment that matters day to day isn't the doomer kind; it's AI optimizing to align its representations with yours, which Perszyk argues is the very method for building AI that increases rather than erodes human agency.

So in a sense, alignment is the solution, not not the problem, for building AI that gives us more agency.

— Danielle Perszyk, Latent Space
Key Insight
Collapsing two meanings of "alignment" is deliberate. By arguing the safety concept and the capability method are the same thing — get the model to align its representations with yours — Perszyk turns alignment from a tax you pay into the mechanism that produces the intelligence you actually want. Marr's 1982 framing is the scaffold: fix the computational goal before the algorithm or the hardware.