Agent Know-How

GitHub found that when a coding agent's output was truncated or summarized to save tokens, the agent sometimes reopened the original or reran the command, turning one cheaper turn into more turns and more context. Its fixes, measured across the whole task:

• Selectively compress repetitive build/test logs while keeping source code (the largest cut)
• Strip unused line-number formatting (5%)
• Less verbose Task-tool prompts (2.9%)
• Deliver background results without an extra retrieval call (2.3%)

Each change passed offline benchmarks and online A/B tests. Measure cost per task, not per tool call.

Codex 0.150.1 makes remote compaction count retained images toward its token budget by default, trimming the oldest images as needed.

Before, accumulated screenshots could inflate the context re-sent every turn without showing up in the compaction budget. Agents that work from screenshots should account for images as tokens, not as free attachments.

"Agentic Context Management" (ACM) argues production agents fail less from weak reasoning than from unmanaged context: histories, large prompts, tool definitions, ballooning tool outputs. It frames three regimes:

• Naive accumulation: per-step cost grows quadratically with conversation length.
• Crude summarization: linear cost, but an accuracy cliff.
• Validated compaction: linear cost with fidelity preserved.

Its reference implementation is Maximem Synap. Treat memory and cost as one lifecycle problem; see context compaction.

Claude Code v2.1.216 fixed a slowdown where long-session message normalization cost grew quadratically with the number of turns, causing multi-second stalls and slow resumes. The same release split filesystem isolation from network egress control into separate sandbox settings, so a team can pay for each control only when it needs it.

The harness's own bookkeeping, not only its model calls, can make a long-running session slow and expensive. Profile the harness, not just the token bill.

A practitioner measured page costs in Claude Code research sessions: an average Wikipedia article is 68,240 tokens of raw HTML (Nike's homepage is 353,000), against about 950 tokens once the web-fetch tool summarizes it.

The cheap path can invert. On JS-rendered or anti-bot pages the fetch returns nothing useful and the agent dumps the raw HTML into context anyway, paying the worst-case bill for a failed read. Fetched content is its own cost line and needs its own guard.

When GitHub moved Copilot code review onto shared Unix-style tools (grep/glob/view), average cost went up. The tools' instructions invited the broad, exploratory browsing that suits an interactive assistant, not the narrow, diff-anchored search a reviewer needs.

Rewriting the instructions to start from the diff, batch searches before reading, and read only the needed line ranges cut average review cost about 20% while holding review quality. A tool's instructions are as much a cost surface as its schema.

Anthropic's guide to steering Claude Code compares seven methods (skills, hooks, subagents, rules and more) by the context cost and the authority of each.

Steering and cost are the same knob seen from two sides. Each instruction added to shape behavior consumes context budget on every turn, so a team should pick the cheapest mechanism that carries enough authority for the rule.

Stanford's DeLM reports cutting multi-agent task cost by about 50% by removing the central orchestrator while keeping target completion rates.

The coordination structure, not only the model, dominates multi-agent spend. Topology is a cost decision to make before choosing models; see orchestration.

Sebastian Raschka surveys how models learn low/medium/high reasoning modes: system-prompt conditioning, RL with per-token cost coefficients, SFT mixing thinking and non-thinking examples, or distilling several reasoning-depth specialists into one model.

Token consumption swings roughly 25-50% across effort levels, and a smaller model at high effort can match a larger one at low effort. Model size and effort have to be tuned jointly, and effort becomes a per-request routing decision based on task complexity.

Coverage of Grok 4.5 puts its coding-agent cost about 80% below a comparable frontier setup, at near-frontier speed, but with a higher hallucination rate.

It is the same cost/reliability trade the boundary-contract and harness-tuning cases make, here left unmitigated. A cheaper model with more hallucinations needs the verification those cases add before the saving is real.

LangChain retuned only the Deep Agents harness (prompts, tool schemas, control flow) around NVIDIA's Nemotron 3 Ultra and matched Claude Opus 4.8's best agent run at roughly 8x lower cost, with no fine-tuning and no bigger model. NVIDIA's write-up of the same work reports benchmark-leading accuracy for the pairing.

Scaffolding investment pays off on every call the harness handles; a bigger model buys quality per call. Both posts come from parties to the result, so validate on your own tasks.

LangChain describes improving agents as data mining: cluster failures out of production traces, fine-tune judge models on them that run cheaper than frontier LLMs, then hill-climb the agent against those evals.

It is the cheap-instrumentation-over-model-swap move applied to evaluation. The judge's cost scales with every trace, so it is the first place a downshift pays back.

Generating many parallel attempts per problem is a reliable but costly way to raise answer quality, and by default the attempts are independent, so compute is wasted on redundant solutions. QuasiMoTTo applies quasi-Monte Carlo sampling to spread attempts more evenly across the solution space.

Agent harnesses reach for parallel sampling when one pass is not reliable enough. Correlating the samples cuts the redundancy tax on that pattern rather than the model price.

"Quantization Inflates Reasoning" shows that low-bit post-training quantization, the standard way to cut inference cost, makes reasoning models emit more tokens to reach the same answer. Final-answer accuracy and per-token latency both miss this hidden test-time cost.

The bill that matters is price per token times tokens spent. Every downshift (smaller model, quantized model, cheaper judge) has to be costed on total tokens emitted in the loop, not the sticker price.

ANMA's author found cheaper models ignore architecture rules. Unguided, Claude Haiku 4.5 violated its constraints in 13 of 19 runs; wrapped in explicit boundary contracts (YAML rules plus CLAUDE.md, hooks, and CI checks) it violated them in 0 of 20.

A bit of contract overhead can make a cheap model reliable enough to take the bulk of the work from a frontier one. This is a small, self-reported benchmark from the tool's author, so rerun it on your own codebase.

LangChain and Fireworks fine-tuned an open model to mine perceived-error signals from production traces, matching frontier-judge performance at roughly 1/100th the cost.

Evaluation is a cost line item too. A frontier judge over every trace is often unaffordable; a small judge trained on your own traces makes continuous judging part of the budget instead of a reason to sample. See LLM-as-judge.

TypeSafe's Jev is pitched as a "System One Model" that only decides, classifies, routes, or scores. It reports running more than 100x faster and more than 200x cheaper than small frontier LLMs on those narrow jobs.

It packages the "most turns don't need frontier capability" finding as a purpose-built model rather than a policy over general models. The figures are the vendor's own; the router or classifier step in a harness is where to test it.

"Bandits in Prod" notes that agents often can only judge a configuration (model selection, retrieval depth, prompting strategy, temperature) by running it on live requests and reading noisy feedback, with no representative validation set. It casts this as online hyperparameter optimization, an infinitely many-armed bandit, and introduces IMABO; its IMOSS policy grows the active set of candidate configurations over time.

The authors evaluate it on classical ML tuning and LLM agent configuration. It automates the per-request choice a team would otherwise hand-write and forget to revisit.

GitHub's Project HydraFusion, a research preview in Copilot, routes each coding step to the model that step needs. In controlled offline evaluations its selective workflows matched or exceeded an Opus 5 baseline while reducing estimated workflow cost.

It is selective multi-model orchestration shipped inside a mainstream coding assistant rather than a benchmark paper. The results are offline and the savings are estimates, so production numbers are still to come.

"Pandora's AI Model Routing Box" treats routing across heterogeneous models, harnesses, and inference settings as efficient allocation under costly value estimation: to route well you must estimate which specialist answers best at the lowest cost, and that estimate itself costs compute.

It gives routing a formal footing instead of a heuristic, and names the hidden cost a router adds: the price of deciding.

Glean CEO Arvind Jain argues that frontier model cost and the popularity of open-weight models are driving organizations toward model routing to control AI spend. Glean improves its own routing with human feedback loops at scale.

It is the buyer-side case for routing: a router whose decisions improve from production feedback, rather than a fixed table, is what lets cheaper models take traffic safely.

LangChain benchmarked NVIDIA's NeMo Switchyard router on 145 agent tasks. Only 7% of turns needed a frontier model; routing the rest to cheaper models cut total cost 74% for six points of accuracy.

Most of an agent's turn-by-turn spend goes to calls that did not need frontier capability. Whether six points is an acceptable price depends on the task, which is why the routing policy needs an eval on your own workload.

Millwright is an open-source, Rust-based, self-hosted LLM router built for cost savings, transparency, and performance. Its author launched it as hosted routers proliferate (Ramp Router, Vercel's AI Gateway) and as OpenRouter faces a possible acquisition.

Owning the router gives a team control over per-request model choice without depending on a vendor's continuity. The trade is operational: provider integrations, updates, and uptime become the team's job.

Opus 5.5 lists at $4/$20 per million input/output tokens (about 20% below Opus 5), with cache reads at $0.20/Mtok (about 60% cheaper). Anthropic's worked task costs:

• A 2.8M-token session costs $11.20 uncached vs. $1.62 at a 90% cache-hit rate.
• A high-effort turn adds about $0.40 of thinking but can avoid a similar-cost retry loop.
• A $0.25 compaction pass pays for itself in about ten turns.

Anthropic says it priced and trained Opus 5.5 for coding sessions that run longer and use more context than at Opus 5's launch.

OpenAI overhauled prompt caching for GPT-6: higher cache hit rates, a dashboard with hit-rate diagnostics, and explicit cache breakpoints that let a team control what stays cached instead of relying on an opaque default.

Explicit breakpoints and diagnostics turn caching from a hope into something a team can engineer and monitor, which matters most for agent loops that re-send a large stable prefix every turn.

Amazon Bedrock AgentCore's new runtime pages memory in on demand and reclaims it when a session idles, instead of billing the full container for the session's lifetime. Snapshotting a warmed environment holds P75 cold start near 2 seconds from a 200MB to a 2GB image, versus 5.4-30 seconds before.

The per-GB rate is higher, but most agents bill lower overall because they stop paying for idle memory. It applies "pay for behavior, not held capacity" to the runtime, not just the model call.

Anthropic's commerce-agents guide reports 90-99% prompt-cache hit rates in production by keeping a byte-identical prefix across three segments: global (rarely changes), session (stable for the conversation), and volatile (changes each turn). Only the volatile segment breaks the cache, and cached tokens read 1.5-2x faster.

Its latency levers are fewer turns, faster tools (dispatching a tool call as its arguments stream), and model choice by eval sweeps over real traffic. Spend-relevant actions stay off the model's say-so: staged, human-approved, with server-issued IDs and transaction caps.

Heidi Health found each ASR inference request used only 15-20% of an NVIDIA L40S's streaming multiprocessors. Packing 4-8 concurrent requests per GPU with CUDA's Multi-Process Service and Triton, instead of one GPU per request, cut the GPUs needed for the same throughput by 75%, from 16 instances to 4.

Utilization, not list price, often sets the self-hosted bill. Measure how much of each GPU a request uses before buying more.

vLLM's Decode Context Parallelism shards the KV cache across GPUs by sequence dimension instead of offloading it, reporting 3x higher throughput on long-context agentic workloads versus standard tensor parallelism.

More throughput per GPU-hour on the same hardware is a direct cost lever, not only a latency one. See agent latency for the serving detail.

Google reports that GKE Agent Sandbox cuts cost per agent by roughly 75% for platform teams running many agents. Agents work in bursts, so one VM per agent (the simple OpenClaw- or Hermes-on-a-VM setup) pays for idle time between bursts.

The sandboxing choice is a cost lever, not only a security control: how agents are packed onto infrastructure sets the per-agent bill. See sandboxing.

OpenLake is an open-source storage engine that moves LLM KV caches from GPU memory into a shared RAM/NVMe tier, compressing blocks losslessly before they leave the GPU. A prefix cached on one host is cheap to fetch from another instead of being recomputed.

On a 128K-context workload, total GPU time fell from 1,169 to 606 seconds, a 48.2% GPU-cost reduction. As agent contexts outgrow GPU memory, KV-cache offload becomes its own storage-engineering problem.

NVIDIA frames infrastructure buying around tokens per dollar and per watt rather than peak chip specs. Its numbers:

• GB300 NVL72 delivers 10-25x the performance per watt of Hopper across three current open models.
• A further 5x software-only gain on one model within a month, from quantization, disaggregated serving, and KV-cache offloading.
• Power-shifting software runs up to 40% more GPUs in the same power budget.

These are vendor figures. For self-hosted agents, the serving stack sets the floor price every other optimization multiplies against.

Open-weight models passed proprietary ones by token volume on Vercel's AI Gateway for the first time: 54% on one day and a record 62% on another, up from 28% on June 24. DeepSeek-V4-Flash was the single most-used model, with Chinese models (StepFun's Step 3.7 Flash, Zhipu's GLM-5.2) filling out the top five.

Price pressure now shows up in where traffic goes, not only in list prices. One gateway's mix is not the market, but it is measured routing behavior rather than a forecast.

Zhipu's GLM-5.3-Flash scores within three points of its larger GLM-5.3 on Artificial Analysis's Intelligence Index at roughly a seventh of the cost, with all inference running on Chinese chips rather than Nvidia hardware.

A small quality gap at a large price gap is the shape that makes Flash tiers the default for routed bulk traffic. Open SWE later used it as its cheapest routing tier.

An FT story on the Ramp AI Index (credit-card billing data from about 70,000 companies) shows Opus 5, then weeks old, capturing 3.5% of Anthropic spend in July, while the prior Opus 4.8 took 28%. Anthropic's annualized revenue still rose to $65bn from $47bn in May.

The spend is real and growing; it just does not concentrate on the newest frontier model. That supports the routing and effort arguments: most per-task spend does not need frontier capability.

Gartner forecasts that inference cost per agentic workflow will increase more than fivefold through 2028, even as per-token prices fall.

Workflow spend scales with the steps and tool calls an agent takes, not with the price of any one token. A falling price list does not guarantee a falling bill once workflows get more agentic.

Reporting has AMD's MI355X undercutting Nvidia's B300 on cost per token to serve Moonshot's Kimi K3.

Hardware competition pushes the open-weight price curve down from the serving side, not only the model side. For self-hosters, the accelerator choice is now a per-token cost decision per model, not a default.

GPT-5.6 cut list price by 20-80%, and the cost of GPT-5.4-level intelligence is reported down roughly 13x in four months, attributed to recursive self-optimization: using the model to distill and improve its successor.

Frontier per-token prices are falling fast enough that a cost model built on last quarter's price list is already stale. Re-price agent workloads on each release.

DeepSeek released another cheaper model as China's domestic price war intensifies (Semafor). A week later, DeepSeek V4 Flash was reported running a full test suite for $72, about 33x cheaper than Kimi K3 on the same job.

The price war is not only cutting list prices; it is widening the gap between individual open-weight releases. Cost per job varies by more than an order of magnitude among models of similar class, so benchmark cost on your own tasks.

Reporting in Memeburn finds Chinese AI models gaining US users on the strength of lower costs and open weights.

It is an early signal of the shift later measured in gateway traffic: price, not only capability, is moving where agent workloads run. Open weights also give buyers a self-hosting option that caps their exposure to any one vendor's price changes.