If every call your agent makes goes to a frontier model, you are almost certainly overpaying for capability most of those calls don't need. LangChain's own benchmark of a production routing system found only 7% of agent turns actually required the frontier model; the other 93% were handled by a 30B model with a 6-point accuracy cost. Gartner is forecasting per-workflow inference cost to rise more than fivefold by 2028, so this stops being a nice-to-have optimization and becomes the difference between an agent that scales and one whose unit economics get worse as usage grows.
"},{"heading":"Short answer","html":"Route per call, not per task type: default every call to a cheap or mid-tier model, and escalate to the frontier model only when a specific, cheap-to-compute signal says the current call needs it. Three independently built production routers — LangChain's benchmark of NeMo Switchyard, Databricks' Unity AI Gateway, and Glean's Waldo — all use this shape and report cost cuts between 30% and 74%. The catch in all three: the classifier or judge model that decides when to escalate has its own cost, and in the LangChain benchmark it consumed over a fifth of the routed spend.
"},{"heading":"Builder model","html":"Stop thinking of model routing as "pick a cheaper model for this kind of task." None of the three production systems here route by task category. They route by call, using a signal computed fresh each time:
\nThe shared mental model: cost sits on the whole escalation path, not just at the model swap. A router that reduces frontier calls but runs an expensive judge on every turn can spend a large fraction of its "savings" running the judge itself.
"},{"heading":"Mechanism","html":"LangChain's escalation mode starts every task on Nemotron 3.5 Lightning (30B parameters) and only escalates to Claude Opus 4.8 after two consecutive negative verdicts from a separate judge model — a design meant to keep single unreliable outputs from triggering an expensive escalation, at the cost of running that judge on every turn. Across 145 multi-step agentic tasks (tau-squared-bench airline, Berkeley Function Calling Leaderboard), 93% of the 6.3 average calls per task never left the cheap model.
\nDatabricks' Smart Routing runs classification once, at session start, rather than per call: a lightweight extractor model reads the task description and produces semantic labels (system components, code evidence type, failure pattern, fix localization, project type), which the router turns into task and language "families." The session defaults to a medium-sized model and moves up only when those labels indicate frontier-level capability is required — trading some per-call precision for a cheaper, one-time classification cost.
\nGlean's Waldo model sits earlier in the pipeline: before any frontier call, it decomposes the incoming query and decides which tools and steps the task actually needs, which is what produces the reported 50% latency cut and 25% token cut independent of which model eventually handles each step.
"},{"heading":"Evidence","html":"See agent cost for the broader problem of runaway agent token spend this concept is one mitigation for, and cost controls for budgeting and per-task attribution techniques that pair with routing rather than replace it.
"}],"evidence":[{"id":"langchain-2026-switchyard-agent-routing-benchmark","kind":"benchmark-result","tier":"benchmark/result-backed","title":"Agent routing benchmark: NVIDIA NeMo Switchyard","note":"LangChain benchmarked NeMo Switchyard's escalation-mode routing across 145 multi-step agentic tasks (avg. 6.3 model calls each, drawn from tau-squared-bench airline and the Berkeley Function Calling Leaderboard). Only 7% of calls needed the frontier model (Claude Opus 4.8); a 30B model (Nemotron 3.5 Lightning) handled the other 93%. Routing cut cost 74% versus Opus-only ($0.026/task vs $0.092/task) for a 6-point accuracy drop (86.0% to 80.0%). The frontier model still consumed 68.4% of spend despite handling 7% of calls, and the judge model used to decide escalation was itself 21.2% of routed spend.","url":"https://www.langchain.com/blog/switchyard-agent-routing-benchmark"},{"id":"databricks-2026-unity-ai-gateway-smart-routing","kind":"primary-doc","tier":"primary-doc-backed","title":"Smart Routing in Unity AI Gateway","note":"Unity AI Gateway's Smart Routing classifies task complexity once at session start with a small, fast extractor model, not a frontier model, then defaults to a medium-sized model and escalates only when the derived task/language labels call for frontier-level capability. Databricks reports 35% cost savings on internal benchmarking, 56% on public benchmarks, and matching Opus 5 quality at under half the cost.","url":"https://www.databricks.com/blog/smart-routing-unity-ai-gateway-match-frontier-quality-30-lower-cost-task"},{"id":"story-b6461cff58b0d468-glean-model-routing","kind":"story","tier":"source story","title":"Frontier Model Cost and Open-Weights Popularity is Driving Demand for Model Routing","note":"Glean CEO Arvind Jain: frontier model prices have doubled to quadrupled release over release, pushing enterprise per-user AI spend up 10-20x year over year. Glean's own pre-filter model, Waldo, decomposes a query and decides which tools it needs before any frontier call, cutting latency 50% and tokens 25%; Glean reports averaging $0.45/task versus $1.84/task for a comparison baseline.","sid":"b6461cff58b0d468"},{"id":"story-c26d5834adc52fbd-gartner-inference-cost-forecast","kind":"story","tier":"source story","title":"Inference Costs per Agentic Workflow to Increase More Than Fivefold Through 2028","note":"Gartner's market forecast: per-workflow inference cost for agentic AI is projected to rise more than 5x by 2028 — the demand-side pressure that makes routing off the frontier model a default architecture decision rather than a one-time optimization.","sid":"c26d5834adc52fbd"},{"id":"agent-model-routing-editorial-synthesis","kind":"editorial-inference","tier":"editorial inference","title":"LLM Digest synthesis","note":"Read together, LangChain's, Databricks', and Glean's independently built routers converge on one shape: classify complexity cheaply before generation, default to a mid-tier or small model, and escalate to frontier only on a specific, cheap-to-compute signal (a judge model's negative verdict, a complexity label, a decomposition pre-filter) rather than by task type or user tier. The three sources disagree on the escalation trigger's own cost and reliability, not on the routing shape — LangChain's judge model alone consumed over a fifth of routed spend, which is the real caution: naive routing can quietly move the cost problem into the classifier instead of removing it."}],"related_topics":[{"slug":"agent-cost","title":"Agent token costs are unpredictable and easily run away"},{"slug":"cost-controls","title":"Cost controls: budgets, metering, and per-task attribution"}],"related_playbook_cards":[],"related_storylines":[],"covers_evidence":["langchain-2026-switchyard-agent-routing-benchmark","databricks-2026-unity-ai-gateway-smart-routing","story-b6461cff58b0d468-glean-model-routing","story-c26d5834adc52fbd-gartner-inference-cost-forecast","agent-model-routing-editorial-synthesis"]}