What are calibrated option probabilities?

A model returns one probability for each answer in a fixed list, and the numbers are **calibrated** when they match real hit rates: among all answers scored at 0.9, about 90% are correct. Microsoft-Decision-1 is built to return scores of this kind in a single pass.

Why does calibration matter for AI agents?

Agents make many small control calls, such as route, retry, stop or hand off. With calibrated scores, a harness can set thresholds once and know in advance roughly how many automatic actions will be wrong, instead of asking a large model to argue its way to each verdict.

Isn't every softmax output already a probability?

Softmax outputs sum to 1, but that does not make them calibrated. A model can score answers at 0.95 that are right only 80% of the time, so a threshold on raw scores can act far more often than you planned.

How is Decision-1 different from using an LLM as a judge?

An LLM judge usually writes reasoning and then a verdict, which is slow and comes with no hit rate. Decision-1 scores all options in one pass and returns the probability as part of the API; Microsoft reports a P50 latency about 35× faster than GPT-6 Sol (self-reported).

Agent· 2026-10-13

Microsoft releases Decision-1 for single-pass agent control — Calibrated option probabilities — What does it mean?

TL;DR

What is it: Microsoft-Decision-1, released on October 9, 2026, is a decision-scoring model that returns calibrated option probabilities: one probability for each answer in a fixed list (yes/no, a category, a rating), where a 90% score is meant to be right about nine times in ten.
Why it’s needed: Agents make many small control calls per task (route, retry, stop, hand off), and a probability intended to match its hit rate lets the harness set a threshold once and act on the number instead of asking a large model to reason its way to a verdict every time.
vs previous: The usual approach is a general LLM acting as a free-text judge, which is slow and gives no number that predicts how often it is wrong; Decision-1 scores every option in a single pass and makes the probability itself part of the API.

Jargon

Calibration: A model is calibrated when its stated confidence matches its hit rate: among all answers it scores at 0.7, about 70% are correct.
Logits and softmax: Logits are the raw scores a model gives each candidate; softmax turns them into probabilities that sum to 1. See From Logits to Probabilities.
Overconfidence: A model that says 0.95 but is right only 80% of the time. The probabilities still sum to 1; they just overstate the hit rate.
Reliability table: Group predictions by score (0.6–0.7, 0.7–0.8, and so on) and count the real accuracy of each group. A calibrated model's groups match their scores.
LLM-as-a-judge: Using a general language model to read an output and write a verdict, often with reasoning text before it.
Decision-flip rate: How often a model changes its answer when the same request is reworded or its options are reordered.
P50 latency: The median response time: half of the requests finish faster than this.
Escalation: Passing a decision up to a stronger model, a tool or a person when the cheap gate is not confident enough.

The news. On October 9, 2026, Microsoft released Microsoft-Decision-1, a post-trained Qwen3.5-9B model for routing, classification, prioritization, verification and workflow control, available in Microsoft Foundry and on OpenRouter. Given a fixed set of options, it returns a calibrated probability for each through a structured API call. Microsoft reports the highest accuracy in its own 36-benchmark comparison (nearly 150,000 questions kept blind from training), a P50 latency about 35× faster than GPT-6 Sol and 2.5× faster than the runner-up H2O-Lightning-4B v1.1, and a price of $0.042 per million input tokens with output tokens free. All of these figures are self-reported. Read the announcement →

Picture two weather forecasters. The first says "probably rain" every morning; you cannot tell a confident "probably" from a nervous one, so you carry an umbrella every day. The second posts a number, and you have checked her record: on the days she says 70%, it rains on about 7 of them in 10. That record is what makes her number useful, because you can write a rule like "umbrella above 40%" and know in advance how often it will let you down. Calibration is that record: a probability is calibrated when, among all the cases scored at p, the share that turn out correct is close to p.

An agent's harness meets the forecaster problem many times per task. Should this ticket route to billing or to support? Did the last tool call succeed, or should the agent retry? Is the work finished, or should the agent stop and ask a person? Each one is a choice from a short, fixed list of options, which is exactly the shape Decision-1 is built for. The caller sends the request and the options, and the model returns one probability per option in a single pass, instead of writing a paragraph of reasoning and then a verdict.

Request + optionse.g. act, retry, hand off

Single passno reasoning text generated

One probability each0.91 / 0.06 / 0.03

Threshold bandsact, review or hand off

Why not read the probabilities that any language model already produces? Every model ends in a softmax over logits, so every model emits numbers that sum to 1. But summing to 1 is not the same as being calibrated: a model can put 0.95 on answers that are right only 80% of the time, and a threshold set on an overconfident score acts too often, so the error budget you planned is not the one you get. Microsoft's framing is that "the probability itself is part of the API": the number is meant to be read as a hit rate, not only as a ranking. The launch page states the target (a 90% prediction should be right about nine times out of ten on representative cases), but it does not print a calibration error figure, and it does not describe the post-training recipe.

Once the number means a hit rate, the umbrella rule becomes a harness policy. A common pattern is three bands: above a high bar, act automatically; in the middle, defer to a retry or a stronger model; below a low bar, hand off to a person. The bars are a business choice, and calibration is what lets you price them before launch, because if the band is calibrated on representative traffic, its expected error rate is roughly one minus its average score. This is the same fail-safe vs fail-open choice a guardrail makes, with a dial instead of a switch. Microsoft also reports a robustness figure that matters for gates: Decision-1 changes its decision on 1.3% of perturbed requests on average across eight perturbation types, and on none when the options are paraphrased, reversed or shuffled.

Where the numbers land (illustrative). Take 1,000 support tickets and a gate with three options: auto-resolve, retry with a larger model, or hand to a person. Suppose 600 tickets score at least 0.9 on auto-resolve, with an average score of 0.95 among them. If the gate is calibrated, about 600 × (1 − 0.95) = 30 wrong auto-resolutions, a number you can compare with your error budget before launch; the other 400 tickets take the slower paths. Now run the same 600 through an overconfident gate whose 0.95 really means 0.85: the count becomes 600 × 0.15 = 90 wrong, three times the budget, with no warning in the scores themselves. Latency adds up the same way. Microsoft's own example is that adding 100 ms to each of 20 sequential decisions adds about 2 seconds to a task, which is why a gate inside an agent loop has to be cheap per call.

Approach	What it returns	Can you set a threshold on it?	Cost per decision
Free-text LLM judge	A verdict, often after reasoning text	Only after parsing, and no hit rate is attached	A full generation with many output tokens
Raw softmax of a general model	Probabilities that sum to 1	Yes, but 0.9 may not mean 90% right	One pass
Calibrated decision scorer (Decision-1)	One calibrated probability per option	Yes; 0.9 is meant to be right about 9 in 10 (Microsoft, self-reported)	One pass; $0.042 per million input tokens, output free (source)

Calibration has to be checked on your own traffic, because the promise holds only on representative cases. A gate tuned on one mix of requests can drift out of calibration on another, so the practical check is a reliability table on a held-out sample of your own decisions: group them by score, count the hit rate in each group, and move the bars when a group drifts. This is Pass/Fail vs Score applied to the gate itself, and it is the step that turns a vendor's calibration claim into a number you can rely on.

Goes deeper in: AI Agents → Workflow Patterns → Chaining + Routing

Related explainers

LLM-as-a-Verifier: continuous verifier scores — reads the whole score distribution instead of one label
HARISSA: hidden-state correctness probes — predicts failure from activations and escalates to a person
EarlyEval: calibrated early stopping — uses calibrated predictions to stop an eval run early

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based