What is the Token-Energy Law?

It is a compact model of GPU-board energy per generated token for dense, full-history decoding. It adds a constant, a weight-byte term, a KV-byte term and a power-times-time term, with coefficients fitted for each model and hardware setup.

Why does batching lower energy per token but not proportionally?

Batching shares the weight bytes, so that term falls as one divided by the effective batch. The private KV term, the constant and the time term do not shrink at the same rate, so total energy per token falls by less than the batch grows.

How does it relate to the roofline model?

Both treat decode as limited by data movement and time rather than arithmetic. The law is described in the paper as following the work-time view of energy roofline modeling, with bytes as the work and realized time as the clock.

GPU· 2026-10-22

Token-Energy Law — Active bytes and decode time — What does it mean?

TL;DR

What is it: The Token-Energy Law is a model from an October 2026 arXiv paper that predicts the GPU energy spent per generated token from active bytes (weight bytes shared by a batch plus private KV-cache bytes) and the realized time per token.
Why it’s needed: Decode is limited by moving bytes, so energy per token is set by how many bytes each token drags through memory and how long the GPU stays busy, which lets you reason about batching, context length and throughput as one energy budget.
vs previous: A plain energy equals average power times duration estimate uses one number per run; the law splits the bytes into a shared-weight term and a per-request KV term, and in the paper's held-out tests a time-only model missed by 10.972% to 29.395% where the full model stayed within 0.887% to 2.137% (GB300 runs).

Jargon

Decode: The phase where the model produces output one token at a time. Each step needs the dense weights and the request's cached history, which the paper counts as its principal byte demand. See Prefill vs Decode.
KV cache: The stored keys and values of every earlier token, so attention does not recompute them. It is private to each request and grows with every token. See Memory Cost.
Active bytes: The paper's count of the bytes a decode step logically needs per output token, split into shared weights and private KV history. It is demand, not the number of times a kernel physically loads a byte.
Realized decode time: Elapsed decode duration divided by the number of output tokens, in seconds per token. It is the inverse of throughput.
Effective batch (B_eff): The average number of useful output tokens produced per decode iteration. It measures executed work, not the configured concurrency limit.
Affine model: A formula of the form constant plus coefficient times one input plus coefficient times another. Here the coefficients are fitted once per model and hardware setup.
HBM: High-bandwidth memory, the GPU's main memory where weights and the KV cache live. See HBM: Where Your Model Lives.

The news. On October 20, 2026, a paper on arXiv introduced the Token-Energy Law, a model of GPU-board energy per output token for dense, full-history decoding. It was tested on three BF16 models on NVIDIA Hopper and Blackwell GPUs, including two-GPU tensor parallelism on Hopper, and held-out condition means were predicted within 3.73%. Read the paper →

Picture the shuttle bus again. Think of the fuel as a cost the whole bus pays for each trip, whoever is aboard, so the more riders share a trip, the less fuel each one pays for. That is the first term of the law: during decode, every request in the batch uses the same weights, so the weight bytes per output token fall as one divided by the effective batch size. Doubling the batch halves that term. The paper is careful that this does not halve the energy per token, because the rest of the bill does not shrink with it. This is the same trade you meet in continuous batching.

Luggage is the second term. Each rider brings a private bag, and the bus must carry every bag on every trip. In a model, each request must read its whole KV cache again for every new token, so private KV bytes per token grow linearly with the attended history. Adding riders does not help here, because nobody shares a bag. This is why a batch of long conversations is expensive even when the batch is large, and why memory limits the batch.

The third term is the meter. Even with the same fuel and luggage, a bus stuck in slow traffic costs more because the meter keeps running. In the law this is a coefficient in watts multiplied by the realized time per token, so a serving setup that finishes each token sooner pays less time-associated energy at the same byte counts. Together the law reads: energy per token is a constant, plus a weight coefficient times shared-weight bytes, plus a KV coefficient times private KV bytes, plus a power coefficient times time per token. The coefficients are fitted once for each model, hardware, runtime and topology, so they describe local energy responses rather than separate physical meters.

Worked example (illustrative). Take a 7-billion-parameter model in BF16, so about 14 GB of weights, and a KV cache of about 0.06 MB per cached token. Hold the average history at 4,000 tokens, which gives roughly 0.24 GB of private KV bytes per output token. At an effective batch of 8, the shared weight term is 14 GB divided by 8, which is 1.75 GB per token, for about 1.99 GB of active bytes in total. At an effective batch of 32 it is 14 GB divided by 32, which is 0.44 GB per token, for about 0.68 GB in total. Active bytes per token fall from about 1.99 GB to 0.68 GB, roughly 2.9 times fewer, while the KV share stays at 0.24 GB. Energy per token falls by less than that factor, because the time term and the constant still apply and the batch changes the realized time as well.

Model (GB300, six held-out cells)	Bytes + time	Time only	Bytes only	Total bytes + time
Qwen2.5-7B	2.137%	29.395%	35.458%	5.721%
Qwen2.5-14B	0.971%	10.972%	28.228%	4.381%
Mistral-7B	0.887%	14.117%	33.564%	5.589%

The table is the paper's own ablation: the the full model, with both byte terms and the time term, gave the lowest maximum cell-mean error in every comparison, under 2.2%. Dropping time or the bytes pushes the error above 10% for all three models. Forcing weights and KV to share one coefficient (the total-bytes column) still passes but is two to five times worse, which supports keeping them separate: their reuse and access patterns differ. The paper also reports that coefficients identified earlier stayed effective when requests left the batch at different times, without recalibration.

What the law does not claim matters as much. It covers dense, full-history decoding in BF16 and measures GPU-board energy only, not the whole server or facility. The coefficients are fitted effective responses, not separately metered components. The evaluation used observed decode time, so predicting energy before running needs a separate, independently validated performance model.

Goes deeper in: LLM Internals → KV Cache → Memory Cost

Related explainers

NVIDIA AI Factories — Tokens-per-megawatt as a serving metric — the fleet-level view of the same energy-per-token question
KV-streams — Keeping the KV cache through context compaction — a way to drop old KV entries in place, which shrinks the private KV bytes a request carries

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based