Agent Builder Foundations

Route per call, not per task type: default every call to a cheap or mid-tier model, and escalate to the frontier model only when a specific, cheap-to-compute signal says the current call needs it. Three independently built production routers — LangChain's benchmark of NeMo Switchyard, Databricks' Unity AI Gateway, and Glean's Waldo — all use this shape and report cost cuts between 30% and 74%. The catch in all three: the classifier or judge model that decides when to escalate has its own cost, and in the LangChain benchmark it consumed over a fifth of the routed spend.

Stop thinking of model routing as "pick a cheaper model for this kind of task." None of the three production systems here route by task category. They route by call, using a signal computed fresh each time:

• LangChain / NeMo Switchyard — escalation mode: every task starts on the cheap model; after two consecutive negative verdicts from a judge model, it escalates permanently to frontier.
• Databricks / Unity AI Gateway — classification mode: a small, fast extractor model labels the task once at session start (affected components, code evidence present, failure pattern, fix scope), and those labels move the session up or down from a medium-sized default.
• Glean / Waldo — decomposition mode: a pre-filter model breaks the query down and decides which tools and steps are needed before any frontier call happens, trimming tokens and latency even before routing a specific step.

The shared mental model: cost sits on the whole escalation path, not just at the model swap. A router that reduces frontier calls but runs an expensive judge on every turn can spend a large fraction of its "savings" running the judge itself.

LangChain's escalation mode starts every task on Nemotron 3.5 Lightning (30B parameters) and only escalates to Claude Opus 4.8 after two consecutive negative verdicts from a separate judge model — a design meant to keep single unreliable outputs from triggering an expensive escalation, at the cost of running that judge on every turn. Across 145 multi-step agentic tasks (tau-squared-bench airline, Berkeley Function Calling Leaderboard), 93% of the 6.3 average calls per task never left the cheap model.

Databricks' Smart Routing runs classification once, at session start, rather than per call: a lightweight extractor model reads the task description and produces semantic labels (system components, code evidence type, failure pattern, fix localization, project type), which the router turns into task and language "families." The session defaults to a medium-sized model and moves up only when those labels indicate frontier-level capability is required — trading some per-call precision for a cheaper, one-time classification cost.

Glean's Waldo model sits earlier in the pipeline: before any frontier call, it decomposes the incoming query and decides which tools and steps the task actually needs, which is what produces the reported 50% latency cut and 25% token cut independent of which model eventually handles each step.

• Benchmark-result-backed (LangChain): a controlled 145-task benchmark of NeMo Switchyard's escalation routing, with exact cost, accuracy, and spend-distribution numbers.
• Primary-doc-backed (Databricks): the vendor's own account of Unity AI Gateway's session-start classification mechanism and reported cost savings.
• Story-backed (Glean, via Latent Space interview): CEO Arvind Jain's account of Waldo's decomposition-based pre-filtering and Glean's per-task cost comparison.
• Story-backed (Gartner, via press release): a market forecast establishing why routing's cost pressure is expected to grow, not just a snapshot of current savings.
• Editorial inference: that these three independently built systems share one underlying shape, and that the escalation/classification step's own cost is the shared risk, is LLM Digest's synthesis across three differently designed routers.

• Route per call, not per task type. A single "coding agent" task mixes trivial file reads with genuinely hard planning steps; routing by task category misses that most of the cost concentrates in a small share of calls within any task.
• Budget the classifier or judge's own inference cost. LangChain's judge model consumed 21.2% of routed spend — a naive routing implementation can silently reintroduce much of the savings you're chasing if the escalation signal itself isn't cheap.
• Pick an escalation signal you can compute cheaply and repeatedly: consecutive judge failures (LangChain), task-complexity labels derived once at session start (Databricks), or a decomposition pre-filter before any frontier call (Glean) — not a static allowlist of task types.
• Decide the accuracy tradeoff explicitly before shipping. LangChain's benchmark traded 6 points of accuracy (86.0% to 80.0%) for a 74% cost cut; that trade is acceptable for some workloads and not others, and it should be a deliberate choice, not a side effect.
• Treat routing as infrastructure that has to keep working as usage grows, not a one-time tuning pass — Gartner's forecast of a fivefold rise in per-workflow inference cost by 2028 is the reason all three vendors here shipped routing as a standing product feature rather than a manual cost-cutting exercise.

• Treating model routing as a one-time model swap (send everything to a cheaper model) instead of a per-call decision with a real escalation path back to frontier capability.
• Ignoring the classifier or judge model's own inference cost, which can consume a fifth or more of total routed spend and quietly erode the savings the router was built to capture.
• Routing on a static rule (task category, user tier, time of day) instead of a live complexity or confidence signal, missing that cost concentrates in a small share of genuinely hard calls regardless of task type.
• Optimizing for cost without measuring the accuracy delta, and shipping a router that trades away more quality than the use case can tolerate.

See agent cost for the broader problem of runaway agent token spend this concept is one mitigation for, and cost controls for budgeting and per-task attribution techniques that pair with routing rather than replace it.