Agent Builder Foundations

Route per call or per thread on a live signal, not per task category. Default to a cheap or mid-tier model and escalate to frontier only when a cheap-to-compute signal says the work needs it. Independent routers at LangChain, Databricks, and Glean use this shape and report 30-74% cost cuts. The shared catch: the classifier or judge that decides escalation has its own cost, and in one benchmark it took 21.2% of routed spend.

Routing is not "pick a cheaper model for this kind of task." Production systems compute a fresh signal and differ mainly in where they compute it:

• Escalation (NeMo Switchyard) — start on the cheap model; escalate to frontier after two consecutive negative judge verdicts.
• Session classification (Databricks Unity AI Gateway) — a small extractor labels the task once at session start; labels move it up or down from a medium default.
• Thread classification (LangChain Open SWE) — one classifier decision per thread picks a fast, balanced, or performance tier.
• Decomposition (Glean Waldo) — a pre-filter breaks the query down and picks tools before any frontier call.

Cost lives on the whole escalation path, not only at the model swap.

Routing works because difficulty is unevenly distributed inside a task. A coding or support task mixes trivial reads with a few hard planning steps, so most calls clear on a small model and cost concentrates in a small share of calls. Switchyard's 93/7 split shows this; the frontier model still took 68.4% of spend on its 7% of calls.

Where the signal is computed sets the tradeoff:

• Per-call judging is most precise but runs the judge every turn. Requiring two consecutive negative verdicts filters noisy single failures, at the price of that recurring judge cost.
• One-time classification (per session or per thread) is cheap but blind to drift. Open SWE's router does not re-route when a thread changes topic.
• Pre-generation decomposition cuts tokens and latency before any model choice, independent of which model handles each step.

Quality has to be measured on an outcome. Open SWE's live A/B showed a 64% lower median thread cost with merged-PR rate statistically unchanged; a fast-only arm was pulled after a day of user complaints. Its tiers came from the cost-versus-capability frontier and its criteria from its own traces, so neither transfers unchanged to another agent.

• Route on a live signal, not a static rule. Use judge verdicts, complexity labels, or a decomposition step; not task category, user tier, or time of day.
• Budget the classifier's cost. Measure judge or classifier spend as a line item; if it approaches the savings, move to one-time classification or a smaller judge.
• A/B the router on an outcome metric. Compare merged PRs, resolved tickets, or task success between routed and control traffic; cost alone hides quality loss.
• Derive tiers and criteria from your own traces. Copying another team's thresholds skips the step that makes routing work.
• Decide the accuracy trade explicitly. Switchyard traded 6 points of accuracy for 74% lower cost; write down whether your workload tolerates that before shipping.
• Re-classify on drift if one-time classification is used and threads change scope mid-run.

• Treating routing as a one-time swap to a cheaper model, with no escalation path back to frontier.
• Ignoring the judge's own cost, which can consume a fifth of routed spend.
• Routing on task category or user tier and missing that cost concentrates in a few hard calls.
• Classifying once per thread and assuming the label holds after the topic changes.
• Reporting cost savings without an outcome comparison, and shipping a router that loses more quality than the use case allows.

See agent cost, cost controls for budgeting that pairs with routing, and replay versus live evaluation of routers for why router accuracy needs live-rollout evidence.