Agent engineering · know-how
🧱 Obstacle·cost·active·9 entries·9 sources·updated 2026-10-06
Calculating the true return on investment (ROI) for agent systems is blocked by the difficulty of measuring time savings, tracking per-task token usage, and accounting for hidden costs like token inflation in quantized models. Platform engineers must move from generic productivity claims to instrumented cost-per-task accounting and evidence-based measurement of outcomes.
The ROI vocabulary is converging on cost per successful task. OpenAI's CFO proposes it as a core scorecard metric, and Anthropic tells buyers to pick models on cost per task, settled by their own evals rather than a leaderboard. Proving ROI means attributing spend and outcome to the same unit of work.
Attribution tooling now reaches that unit. Open-source meters attribute tokens to a pull request or a coding agent. Vendor consoles, self-hosted gateways, and platform governance hubs track spend per user and project and surface untagged spend, the gap that breaks chargeback.
The cost side has hidden traps. Per-token savings can vanish in total tokens: quantized reasoning models emit more of them. Measured splits show how much spend buys nothing: only about 7% of agent turns needed a frontier model, and dropping a central orchestrator cut multi-agent cost about half.
The outcome side is still mostly vendor case studies. Reported numbers include a 21% engineering productivity lift and a 250% conversion lift with 40 hours saved per rep. Each is a single, vendor-published customer figure.
The counter-example is the most useful data point. A frontier coding-model rollout to about 3,500 engineers raised total coding spend about 60%, because the model helps on complex work but not the medium- and low-complexity majority. The fix was a budget tier steering the model to tasks where it pays off: cost-per-task attribution, applied after the fact.
The open problem is independent, per-task instrumentation on the outcome side. Spend is now measurable per task; value delivered per task mostly is not.
Platform engineers cannot justify AI budgets on vague productivity claims. They need instrumentation that tracks cost per task, measures outcomes against the labor they replace, and stops token runaway.
When evaluating a model downshift, routing policy, or quantization, cost it on total tokens consumed in the trace, not the sticker price per token. Roll out expensive models by task tier, with attribution in place first, rather than blanket deployment. See cost controls and agent cost.
Session-level meters, self-hosted gateways, and platform governance hubs push spend attribution below the account aggregate, the data a cost-per-task case needs.
Databricks' Governance Hub layers an AI-specific view on its account-wide cost dashboards. It tracks token consumption and model activity through Unity AI Gateway, flags per-user spend against budget thresholds, and surfaces untagged spend.
Untagged spend is what breaks chargeback. Pushing attribution down to the user and project level gives the cost-per-task framing the data it needs, instead of an account-wide aggregate.
The Claude apps gateway for AWS is a self-hosted control plane that gives an org one point of control over access, cost, and policy for Claude Code and Claude Desktop, running on Amazon Bedrock and Claude Platform on AWS.
It moves governance past a vendor's own admin console into infrastructure the org runs itself. The trade is the usual one: more control over attribution and policy, plus a service to operate.
AgentMeter, an open-source tool, tells a developer what their AI coding agents cost, breaking token spend down by agent rather than leaving it in one provider bill.
Session-level attribution is the raw material for an ROI case. Without it, the only number available is aggregate API spend, which cannot be set against any single outcome.
Finance and model vendors now frame ROI and model choice as cost per successful task, settled by workload-specific evals rather than per-token price or leaderboards.
Anthropic's model-selection guide tells buyers to compare model classes on cost per task, not cost per token, then settle the choice with evals built for their own workload rather than a leaderboard score.
It folds model selection into the ROI calculation. A cheaper per-token model that needs more tokens or retries can cost more per task, and only a workload-specific eval shows which way it goes.
Sarah Friar, OpenAI's CFO, proposed a practical AI scorecard built on four measures: useful work delivered, cost per successful task, dependability, and return on compute.
It is the per-task attribution engineers argue for, pushed by a finance function. That suggests cost per successful task is becoming the shared ROI vocabulary, not one platform team's convention. It also arrives from a vendor that benefits from compute spend, so treat it as a framing, not a neutral standard.
Vendor-published case studies report productivity and conversion gains, but a 3,500-engineer rollout that raised spend ~60% shows unattributed deployment can erase them.
Databricks reports that rolling out a frontier coding model (Astra, per the newsletter headline) to roughly 3,500 engineers raised total coding spend by about 60%, despite the model's reputation for token efficiency. The model "unambiguously" helps on complex, long-horizon tasks but "may not materially improve medium/low-complexity coding," the bulk of daily work.
Databricks responded with a dedicated budget tier that steers the model toward tasks where it pays off. The figure reached the public secondhand, via an executive's post quoted in a newsletter, so treat it as directional, not audited.
LangChain's own sales agent, in production since December 2025, reports:
The same vendor-published, single-customer caveat as other case studies applies. The useful part is the metric set: conversion lift, hours reclaimed, and adoption rate together say more than one productivity percentage.
1Password reports a 21% engineering productivity lift from adopting Codex, while keeping its existing security review policies in place.
It is an outcome-side number rather than a spend number. It is also a vendor-published, single-customer figure: a data point, not the independent per-task instrumentation a team needs to make its own case.
Simon Willison collects anecdotes of people using coding agents to reverse-engineer undocumented home-device APIs and automate them. Before agents this was possible but rarely worth the maintenance risk. When writing the automation is nearly free, so is throwing it away and rewriting it when the API changes.
The ROI shift is not only a lower per-task bill. It removes the cost that used to gate the work, so the set of things worth building grows.
Also relevant to Agent token costs are unpredictable and easily run away