Small routers switch LLM capabilities per query — Independent capability gates vs single-choice routing — What does it mean?
The news. On November 12, 2026, a paper from CORTEXA Innovations, Small Language Models for Inference-Capability Routing, compared Qwen2.5 routers at 1.5B, 3B, 7B and 14B parameters under five prompting strategies. It then fixed the best prompt per capability and ran 127 questions through a fixed GPT-OSS-120B downstream model under seven conditions: all capabilities off, all on, an oracle that used the benchmark labels, and the four routers. Read the paper →
Picture a house at night with one wall switch per room. Someone walks in and flips only the switches they expect to need: the study, maybe the kitchen, not the attic. A capability router does the same for each incoming question: switch by switch, it decides which optional capabilities the big model gets. In this paper the switches are concrete. The reasoning switch sets the downstream model's thinking budget (reasoning_effort) to high instead of low. The retrieval switch runs RAG and adds retrieved passages to the prompt. The web-search switch adds search results. The calculator switch only makes the tool available — the downstream model still decides whether to call it.
The design choice that matters is that the four switches are independent. A routing step in a workflow usually sends a request down one branch, like a dial that can light only one room. That fits a question that needs either retrieval or extra thinking, but not one that needs a document lookup and a calculation. Independent gates replace one multi-way choice with several yes/no choices, so any combination can be switched on for the same question. The arithmetic is simple: four yes/no gates cover 2 × 2 × 2 × 2 = 16 combinations with only four small decisions, while a single dial would need a separate position for every combination. The paper also picks a safe default for broken router output: if a gate's answer cannot be parsed, that gate is switched on, so a router failure can add cost but cannot silently remove a capability.
Independence also changes what a mistake costs. A light left burning in an empty room is a false positive: the downstream model pays tokens and seconds for a capability the question did not need. A dark room that someone needed is a false negative: the answer may lose the evidence or computation it required. The two errors land on different bills — false positives on cost, false negatives on quality — and a single F1 score blends them into one number. The paper's own benchmark shows how misleading that can be: 83 of its 127 questions are labelled as needing reasoning, so a router that always answers yes would get a reasoning F1 of about 0.790 — higher than every trained router — while paying for high reasoning effort on all 44 questions that did not need it.
| Condition | Mean BERTScore F1 | Avg downstream tokens | Median downstream latency | Source |
|---|---|---|---|---|
| All capabilities on | 0.8418 | ~8,837 | 9.79 s | paper §5.4 |
| Oracle (benchmark labels) | 0.8411 | 2,977 | 2.66 s | paper §5.4 |
| 14B router | 0.8408 | 2,764 | 1.91 s | paper §5.4 |
| 7B router | 0.8395 | 2,369 | 3.05 s | paper §5.4 |
| 3B router | 0.8335 | 2,044 | 1.97 s | paper §5.4 |
| 1.5B router | 0.8390 | 4,746 | 2.84 s | paper §5.4 |
Where the tokens actually go
Hold the downstream model (GPT-OSS-120B), the 127 questions and the scoring method fixed, and compare the 1.5B and 14B routers. Their mean BERTScore F1 is almost the same: 0.8390 vs 0.8408. Their switch patterns are not. The 1.5B router produced 55 web-search false positives and 65 calculator false positives, lighting empty rooms on question after question. The result is 4,746 vs 2,764 downstream tokens per question — 1,982 extra tokens per question, or about 252,000 extra tokens across the benchmark (1,982 × 127). Now take the opposite failure. The 3B router was cautious and used the fewest tokens of any router (2,044), but it skipped retrieval on 78 of the 97 questions labelled as needing RAG — about 80% — and it had the lowest routed BERTScore (0.8335). Two cheap-looking routers, two opposite mistakes: one wastes compute, the other starves the answer.
Which switch is hard to flip depends on the capability. Prompt optimization (DSPy few-shot prompts and the GEPA optimizer) narrowed the gap between small and large routers on the reasoning and calculator decisions, but retrieval and web search still gained more from a larger router, and web search was the weakest decision at every router size. The switches are not free either. The router is a model call that runs before the answer, and a bigger router takes longer: median router latency rose from about 1.25 s for the 1.5B router to 5.56 s for the 14B router. The routers ran locally in Google Colab and the downstream model ran on AWS Bedrock, so the authors do not add the two latencies together — the 68.7% token saving describes the downstream model only, not total system cost.
Read the quality column with the same care. BERTScore measures how close an answer is to a reference answer, not whether it is correct, and the all-capabilities-off condition also scored close to the others. The authors say they cannot tell whether the capabilities helped little on this benchmark or whether the metric missed the factual and numerical gains they should bring. The durable lesson is the evaluation method: judge a capability router by its cost profile and by its false-positive and false-negative counts for each capability, measured end to end.
Goes deeper in: AI Agents → Workflow Patterns → Chaining + Routing
Related explainers
- HDM hybrid decision model — Value-of-thinking routing — a single think-or-answer gate, priced per request
- Dynamic LLM Routers are Often Misguided — Cost-matched random routing baseline — another way router scores can flatter a router
- RECAST — Adaptive evidence routing between retrieval and computation — routing between retrieval and computation step by step