The model with the higher token price cost about 23% less per passing answer — Cost per successful outcome — What does it mean?

The news. On September 11, 2026, AWS published results from an open-source harness that runs five OpenAI models down one identical code path: three GPT-5.6 configurations on Amazon Bedrock (luna, terra, sol) and two cost-efficient OpenAI API baselines (gpt-5.4-mini and gpt-5.4-nano). Rather than rank them by dollars per million tokens, it divides total spend by successful outcomes across three workloads — single-call benchmarks, live multi-turn web research, and rubric-graded professional deliverables. The Bedrock models ran with reasoning disabled and the API baselines at their defaults, so AWS frames it as a comparison of deployment configurations, not of intrinsic capability. Read the AWS post →

Picture the claw machine again. The marquee says a dollar a play, and that is the only price posted anywhere on the cabinet. But you did not come for plays. You came for a toy, and the machine charges you the same dollar for the three grabs that came up empty as for the one that did not. Cost per successful outcome is the price you actually paid: everything you spent, across winning and losing attempts alike, divided by the outcomes you kept.

A weak claw moves that number without touching the marquee. AWS reports that gpt-5.4-mini solved 37% of the AIME competition-mathematics problems against sol's 75%. If attempts were independent and accuracy stayed constant, 37% would imply roughly 2.7 plays per success — and the post is careful to add that real retries are correlated, so measure your own retry strategy rather than transplanting that figure. A model that is cheaper per token can still be more expensive per outcome, because the sticker price gets multiplied by how often the model is wrong. The study finds the same thing comparing nano with luna: nano's nominal token price is slightly lower than luna's, yet nano's observed cost per passing research answer was $0.07 against luna's $0.05.

"Successful" has to be defined before anything can be divided by it. The harness sets a different bar per workload: a correct answer on AIME, an F1 of at least 0.7 on the research questions, and at least 70% of weighted rubric points on GDPval. Nor is the bar self-enforcing: grading combines a deterministic pre-pass with a frozen gpt-5.5 autorater that is not one of the models under test. Where you put that bar is not a detail — moving it can change the ranking, which is the same reason the curriculum keeps a pass/fail gate separate from a graded score.

The agent case adds a twist no arcade would get away with: each play costs more than the last. The harness runs with client-managed history and store: false, so the application holds the conversation and re-sends it every turn — system prompt, prior tool results, accumulated search snippets. Each turn is one tick of the agent loop, and turn six pays for turns one through five all over again. Turn count is therefore a pricing variable, and it appears on no pricing page. AWS measured mini at 7.6 turns per research question, the most of any model tested, mostly re-search loops; that pushed it to 114,000 input tokens per question against terra's 50,000, about 2.3×. Terra's higher token price was more than covered by the shorter trajectory: $0.31 per passing answer to mini's $0.40, about 23% less, at a mean F1 of 0.50 against 0.39. This is the cost face of a lesson the agents track teaches as a capacity problem — context is a scarce resource.

Why a shorter trajectory saves more than it looks like it should

Hold one quantity fixed and walk it through. Say every turn adds the same 15,000 tokens of new context — a system prompt plus one round of search results (illustrative). With client-managed history, turn 1 bills 15k, turn 2 bills 30k, turn 3 bills 45k, because each turn re-sends everything before it. Over N turns the bill is not N × 15k but 15k × N(N+1)/2, a triangular pile. At eight turns that is 15k × 36 = 540,000 billed input tokens; at five turns it is 15k × 15 = 225,000. Dropping from eight turns to five cuts the turn count by 37% but removes about 58% of the billed input — which is why AWS writes that a model finishing in five turns instead of eight "can save more than the 37 percent reduction in turns alone suggests." The 15,000 is illustrative and it cancels out of the ratio, so the 58% does not depend on it; only the turn counts do. AWS's own runs sit near this shape rather than on it: it describes cumulative billed input as growing approximately quadratically with turn count, because per-turn context itself only grew roughly linearly.

Put the three workloads together and the shape of the decision changes. Accuracy sets how many attempts a success costs; token efficiency sets what each attempt costs; trajectory length multiplies both. None of the three is printed on a pricing page. Define the outcome and the quality bar your application needs, then divide total spend — failures included — by the outcomes that clear it. The obvious lever once you can see the number is to make the re-sent prefix cheaper rather than shorter, which is what prompt caching is for. AWS's own caveats apply to every figure above: samples run from 48 to 198 items, the Bedrock models ran with reasoning off, and prices moved during the measurement window, so treat the numbers as a worked method rather than a leaderboard. The harness is open source, and swapping in 50 to 100 of your own tasks with known-good outputs produces cost-per-success numbers for your domain: openai-on-aws/benchmarks-openai.

Goes deeper in: Agent Engineering → Cost & Latency Engineering → The Cost Profile of an Agent

Related explainers

• Cache-preserving system messages — the mechanism that keeps a long re-sent prefix eligible for the cache discount, which is what makes those repeatedly billed input tokens cheaper.
• Tokens per megawatt as a serving metric — the same move one layer down: normalize cost by useful output instead of by peak capability.
• LLM judge as advisor — what to do about the fact that a model decides which of your outcomes counted as successful.