TWIML AI Podcast

Damian Borth on Training Models From Models, Not Data

Damian Borth· Professor of AI and Machine Learning at University of St. Gallen

2026-07-27·~46 min·English·TWIML

TrainingLLMAI InfrastructureOpen Source

TL;DR

St. Gallen's Damian Borth argues the weights of already-trained neural networks are a new data modality you can learn from and generate — letting you train new models from old models instead of scarce data, and one day sample a network on demand.

01The Reframe

Weights Are a Modality

<strong>The weights of a trained network are not just the leftover output of training — they are a new input modality</strong> you can learn from, the same way models learn from text and pixels.

The language/pixels recipe, applied to model weights

weights are exciting uh not only as the output of learning but as the input for learning
— Damian Borth, TWIML AI Podcast

Key Insight

Every published model is thousands to millions of GPU-hours of discovered structure sitting idle. Reframing weights as an input modality says that stored compute is reusable raw material, not exhaust — which quietly reclassifies Hugging Face from a download mirror into a training corpus.

02The Origin

If Everything Is Different, Nothing Is Different

The idea began as a software-style question — can you <strong>diff a neural network the way you diff code</strong>? — but raw weights are too unstable to compare, so the fix was to compress a whole population of models into a structured latent map.

Raw weights are unstable; an autoencoder compresses them into a readable manifold

if everything is different nothing is different
— Damian Borth, TWIML AI Podcast

Key Insight

The breakthrough wasn't a bigger model but a change of coordinates. Weights in their raw form have no stable landmarks, so nothing can be measured against anything; projecting a population of models into a low-dimensional manifold is what turns 'a pile of numbers' into a space where accuracy and training history become directions you can navigate.

03Reading Weights

Testing a Model Without Test Data

Once models live on that map, you can <strong>predict a network's accuracy straight from its weights</strong> — no test set, no forward passes over real data, just its position in the latent space.

Encode an unseen network, read its predicted accuracy off the manifold

Can I test the neural network without the use of test data?
— Damian Borth, TWIML AI Podcast

Key Insight

This is the 'analyze' half of the modality claim, and it inverts a core assumption: evaluation normally requires holding out data and running the model on it. If the weights already encode how well a model generalizes, then quality is a readable property of the artifact — useful for triaging the ~30% of Hugging Face models that ship with no documentation at all.

04The Hard Part

The Blurry Weights Problem

Generating weights is harder than reading them: the decoder hits a <strong>low reconstruction error yet still produces a broken network</strong>, because averaging away the tiny high-frequency details throws out exactly the values that make a model work.

Low mean-squared error, working average, broken model

We pluck the weights back to the neural network. Totally screwed up the entire new network.
— Damian Borth, TWIML AI Podcast

Key Insight

The failure is a loss-function story, not a capacity story. Mean-squared error optimizes the average and is blind to the sparse, high-frequency values that carry a network's function — the same trap that made early image VAEs blurry until perceptual losses arrived. It's why generated models still need a short fine-tune to become usable, and why the loss, not the architecture, is the open frontier here.

05The Payoff

Train From Models, Not From Data

Because fresh data is running short and the knowledge is already baked into existing models, you can <strong>train from those models instead</strong> — one remote-sensing result matched a 12,000-GPU-hour baseline in about 350 GPU-hours.

Weight-space learning vs. a from-scratch baseline, in GPU-hours

this is suddenly interesting because you train from models and not from data
— Damian Borth, TWIML AI Podcast

Key Insight

This is the pointed answer to the data wall. The mainstream responses — synthetic data, test-time compute — try to manufacture or stretch data; weight-space learning sidesteps the question by treating the compute already spent on published models as the resource to recycle. A 20–30× compute reduction on a real task is the first evidence that 'reuse the weights' is more than a slogan.

06The Unlock

Prompt With Your Data, Not a Model

The method needed a trained model as a starting anchor — a chicken-and-egg trap — so the next step is to <strong>prompt with a private embedding of your data set</strong>, letting a bank or hospital get weights back without any individual sample ever leaving.

A data-set embedding as the prompt, keeping raw data private

you give this embedding to us, we provide you the weights, we give you the weights, you're much faster in continuing training
— Damian Borth, TWIML AI Podcast

Key Insight

Swapping a model-prompt for a data-set-prompt does two things at once: it escapes the circular need for a good model to make a good model, and it reaches the data that regulation keeps off public repositories. The privacy is structural, not bolted on — if the interface is a single aggregated embedding, the individual records are never in the request to begin with.

07The Vision

Sample a Neural Network on Demand

The endgame is one big foundation model of neural networks from which you <strong>sample the exact model you need in a forward pass</strong> — a ViT, a ResNet, an EfficientNet — replacing pre-training rather than repeating it.

One foundation model of networks, sampling any architecture on demand

we would be actually replacing and totally like replacing pre-training right so why you have pre-trained model you just sample the model that you need
— Damian Borth, TWIML AI Podcast

Key Insight

This reframes model creation as generation rather than optimization. It is complementary to neural architecture search — NAS chooses the structure, weight-space learning fills in the parameters — and because sampling a model is a single cheap forward pass, it opens moves that training can't afford: whole ensembles at once, or a bespoke edge model per device, generated on the spot.

08The Human Arc

You Can't Be Three Times Lucky

None of this looked inevitable: the idea was dismissed as <strong>esoteric and even “useless,” each early result waved off as a toy</strong>, and it only became a field once scattered researchers recognized each other and named the modality.

The five-year path from an esoteric idea to a workshop-backed field

you're lucky twice, but you know, three times, you know, your karma is gone for the next years
— Damian Borth, TWIML AI Podcast

Key Insight

A UC Berkeley collaborator first called the work 'interesting but useless,' and a conference student's badge reading 'weights are the new modality' was an early sign the idea had escaped one lab. The lesson is about how fields form: a dismissed, esoteric bet becomes real not through a single result but when enough independent people converge on it and agree on a shared language.