AI Factory Insider

NVIDIA Engineers on Why Agents Scaled Only After They Got a Sandbox

Nic Borensztein and Jon Fernandez· Distinguished Solutions Architect and Director of Compute at NVIDIA at NVIDIA
·~27 min·English·NVIDIA
AI InfrastructureAgentsInference
TL;DR

Two NVIDIA engineers walk through the AI factory they built for themselves — a slow, long-lived stack underneath, a workload layer that churns on top, and a secure workspace in between that finally let agents run unattended, while internal demand grew to four trillion tokens a month.

01Core Mental Model

The Five-Layer Cake Is Cut in the Middle

Nic Borensztein uses Jensen's five-layer cake to place the split: the reference architecture covers energy, chips and infrastructure, and the validated design covers everything that runs above them.

it would be as a five-layer cake, this is an analogy that Jensen has used that it goes from energy, chips, and infrastructure up to models, applications, and the enterprise reference architecture kind of covers those bottom three layers while the validated design covers everything above

— Nic Borensztein, AI Factory Insider
Key Insight
The cut isn't arbitrary — it separates two very different clock speeds. Below the line sits hardware that Nic later says takes six to nine months just to order, plan and power. Above it sit agent harnesses that his own colleagues rebuilt from scratch twice. Drawing the architecture at that seam lets the fast layer churn without disturbing the slow one.
02The On-Prem Decision

Compliance Pushes You On-Prem; Tokenomics Pulls You

Nic sees two distinct motivations bringing enterprise AI compute on-prem: regulated data that cannot move freely, and token budgets that only become visible once a POC succeeds and scales.

one of the other reasons, increasingly, is simply tokenomics. As people are moving their AI projects from being POCs, that they're just trying to understand the utility, to now they found the utility, they're scaling it up, and now they're getting a look at the token budgets

— Nic Borensztein, AI Factory Insider
Key Insight
Tokenomics is a success problem, not a failure problem — it only bites after the POC proves out and the volume arrives. That timing is awkward: the moment a project earns the right to scale is also the moment its economics force a platform decision. Compliance, by contrast, is knowable on day one, which is why it tends to produce calmer architectures.
03Internal Proof

Chip Nemo Got Rebuilt From Scratch. Twice.

NVIDIA's internal chip-design agent, Chip Nemo, has been in the works for more than three years, and its team has torn the harness down and rebuilt it from scratch twice as the cutting edge kept moving.

But then they redesigned and rebuilt it from scratch. And then the cutting edge shifted again, they redesigned and re-implemented from scratch

— Nic Borensztein, AI Factory Insider
Key Insight
Chip Nemo exists because off-the-shelf agents couldn't do the job: Nic notes the domain runs on non-public internal vocabulary and proprietary file formats. That's the general shape of a defensible internal agent — the moat isn't the harness, which gets thrown away, but the vocabulary and formats nobody outside can train on. Which is also why the rebuilds were survivable.
04The Killer App

The Killer App Was a Desktop That Never Sleeps

Jon Fernandez's team already ran cloud-hosted desktop compute for developers, and pointing it at the need for a protected place to run agents produced a service that thousands of NVIDIA employees now use daily.

It runs twenty-four seven because it's a virtual type of a desktop. It's not a laptop that you shut down and then the work stops. And then secondly, it allows agents to run autonomously, which is the real superpower of agents

— Jon Fernandez, AI Factory Insider
Key Insight
Note what wasn't on a roadmap. The service was sized for roughly a thousand virtual workstation users; once people realised it also worked for autonomous agents, the question became how to fulfil twenty-five thousand requests. A sizing miss that large usually means the demand was already there and simply had nowhere to run.
05The Breakthrough

Approval Fatigue Is a Scaling Wall

Jon frames the old way of working with agents as an endless string of confirmation prompts that both wears the human down and stops the work, which is why running autonomously but safely was the breakthrough.

historically you would have to hit the enter button all the time, right? And so you're working with AI, you're working in some agent. Are you sure you want to do this? Yes or no? Yes, yes, yes, yes, yes. And you get that fatigue, but it also just stops your work

— Jon Fernandez, AI Factory Insider
Key Insight
This quietly reframes what safety costs. Confirming every action is safe but caps an agent's throughput at human attention — and a human answering yes five times in a row isn't really reviewing anything anyway. Spending the safety budget once, on the perimeter, is what lets Jon say the worst case is a task that doesn't work out rather than damage to adjacent property.
06Agent Architecture

Isolation Belongs Next to the Tool Calls

Nic breaks the agent into a perceive-reason-act loop and places the isolation layer alongside the tool calls, the step where an agent actually reaches the outside world.

there's like this perceived reason act loop right yeah perceive is all the context engineering retrieval and memory that goes into it reasoning is the LLM wherever you're running your actual token service and act is all the potential tool calls that you need to make

— Nic Borensztein, AI Factory Insider
Key Insight
Watch which step carries the boundary: Act, where the model's output turns into something that touches the world. Nic's answer to that is VM isolation, a network perimeter, and policy enforcement around where the agent actually runs — safety as a property of the execution environment. That's a durable place to put it, because it holds even as the harness and the model underneath keep changing.
07Scale

Demand Compounds Faster Than You Can Build

NVIDIA's internal token demand grows forty percent month over month while expanding the AI factory takes six to nine months of ordering, planning and power, so the two have to be planned in parallel.

We're serving four trillion tokens a month at ninety nine point nine nearly availability, like two hundred million infrastructure inference requests per day

— Nic Borensztein, AI Factory Insider
Key Insight
Notice that these two clocks are not just different speeds, they are different kinds. Demand compounds monthly; capacity arrives in discrete, months-long steps gated by power and lead times. So capacity specified against today's demand lands into a demand that has moved on, which is why Nic says the two have to be planned in parallel rather than in sequence. The hybrid posture isn't a hedge on architecture; it's what covers the interval a purchase order cannot.
08What's Next

Confidential Compute Could Bring Frontier Models On-Prem

Nic expects confidential computing for inference to catch on, because it gives a frontier lab a way to deploy encrypted weights on-premises while the enterprise keeps custody of its own payloads.

there's this new technology, NVIDIA Confidential Computing, that can provide a secure way for a frontier AI lab to deploy on-premises. So now you have the benefit of this multi-billion dollar IP investment is encrypted in a way that is provably impossible to crack on-prem

— Nic Borensztein, AI Factory Insider
Key Insight
This is the same boundary move as the agent sandbox, applied to the model itself. Nic hedges — he says he thinks frontier models will eventually deploy on-prem — but the mechanism is worth noting: it turns trust into a property of the silicon rather than of a contract. If it lands, what keeps the best models in the cloud stops being a technical constraint.