Agent Builder Foundations

No, not by itself. A 2026 continual-learning study on Terminal-Bench 2.0 found that one popular agent-optimization method actually performed worse than an unoptimized baseline once transferred to new tasks, and a second method stopped improving after its first optimization round — only a method built with explicit regression control kept improving across rounds. Separately, a real 241-turn production coding session surfaced failure modes — an unverified assumption compounding across six build phases, a confidently wrong claim about the codebase, quietly deferred review fixes — that a short, single-task benchmark run is too brief to ever exercise. And Anthropic's own infrastructure-noise study found that, holding the agent completely fixed, moving between resource configurations swung Terminal-Bench 2.0 scores by 6 percentage points — a gap on the same order as what separates top models on public leaderboards. A benchmark score answers "did it pass this fixed set of tasks once, on this specific infrastructure"; production reliability asks "does it keep working as tasks, optimization rounds, session length, and deployment infrastructure change," which is a different, harder question the benchmark score was never designed to answer.

Think of a benchmark run as a single frame from a video: it is accurate for that frame and tells you nothing about what happens in the next one. Three forces move the video forward that a single frame can't show:

1. Re-optimization drift. Teams tune agents against benchmarks repeatedly — a new harness setting, a new fine-tune, a new prompt — and each round is itself an optimization step. A gain measured once is not guaranteed to hold, or even stay positive, after the next round runs against tasks the previous round never saw. 2. Horizon compounding. A benchmark task is usually short: minutes to an hour of agent turns. A production session can run for hundreds of turns. Failure modes that are individually cheap — one wrong assumption, one deferred fix — compound when nothing forces a checkpoint before the next 50 turns build on top of them. 3. Infrastructure confounding. A benchmark score is produced on one specific resource configuration (CPU, RAM, kill thresholds), and Anthropic's own measurement shows that configuration alone moves the score by as much as the gap between different models. A leaderboard comparison that doesn't control for this is comparing infrastructure as much as it's comparing capability.

All three forces mean the fix is the same: measure your own agent's trajectory across rounds, across long sessions, and across the infrastructure you'll actually deploy on — using your own production traces and your own resource budget — instead of reading a single external benchmark number as if it were a permanent, infrastructure-independent property of the agent.

A benchmark score is produced by running an agent (optionally after some optimization step) against a fixed set of tasks once, then reporting the aggregate pass rate. Two structural gaps separate that number from production reliability.

The optimization-gain gap. The Terminal-Bench 2.0 continual-learning study ran three optimization methods through two phases with identical budgets: an initial optimization phase, then a second phase against new tasks, simulating what actually happens when a deployed agent gets re-optimized as new failures surface. GEPA's optimized agent regressed below the unoptimized baseline in the new-task phase — its first-phase gain did not transfer. Meta Harness transferred its gain but plateaued, gaining nothing from a second optimization budget. Only RELAI-VCL, which explicitly checks for regression as part of its optimization loop, kept improving across both phases, reaching a 76.4% lifelong pass rate against a 58.7% baseline. A single-round benchmark score cannot distinguish these three outcomes from each other — all three could report a similar-looking first-round number.

The horizon gap. A benchmark task ends; a production session doesn't, and the failure modes that matter at hundreds of turns are different from the ones a short task exercises. The 241-turn Claude session write-up documents this directly: the agent stated a specific factual claim about the codebase (how many lifecycle hooks existed) that was simply wrong, deferred code-review findings instead of resolving them, and built a six-phase feature on an assumption nobody had verified — all failures that a single short task, scored once, would not have had the length to produce or the structure to catch.

The infrastructure gap. A benchmark score conflates model capability with the resource configuration it happened to run on, and the two are not easy to separate after the fact. Anthropic ran Terminal-Bench 2.0 across six resource configurations with identical Claude models and task sets: the gap between the most- and least-resourced setups was 6 percentage points, and infrastructure-caused error rates ranged from 5.8% under strict enforcement down to 0.5% under uncapped resources. Past roughly 3x resource headroom, additional resources bought almost no further reduction in infrastructure error (1.6 points) while success rates still climbed nearly 4 points — evidence that a meaningful share of the remaining spread past that point is noise from how tightly the harness constrained the agent, not a capability difference. A smaller, still-measurable version of the same effect (1.54 points) showed up on SWE-bench purely from varying RAM allocation. A leaderboard gap between two agents or models that doesn't control for infrastructure configuration may be measuring who got more compute headroom, not who is more capable.

Closing all three gaps takes the same underlying move: mine your own production traces for the specific failure categories you actually see, the way LangChain built IssueBench around 15 named failure categories across three domains instead of relying on a generic pass/fail benchmark; re-check any optimization gain after the agent is re-optimized again rather than trusting the number from the first round; and hold infrastructure configuration constant (or explicitly report it) when comparing scores, calibrating resource limits to roughly 3x headroom the way Anthropic recommends, so the score reflects capability rather than compute budget.

• Benchmark/result-backed: the Terminal-Bench 2.0 continual-learning study shows two of three tested optimization methods either regress or plateau on new tasks, while a regression-aware method reaches a 76.4% lifelong pass rate versus a 58.7% baseline — evidence that a one-round optimization gain does not predict what happens on the next round.
• Production field-report-backed: LangChain's trace-mining loop produced a measured 13.7% harness lift on Terminal-Bench 2.0 and a cheaper fine-tuned judge model that matched frontier-model performance on a narrow task — both derived from mining the team's own production traces, not from a published benchmark score.
• Production field-report-backed: a real 241-turn Claude coding session surfaced a wrong factual claim, deferred review fixes, and an unverified assumption that compounded across six build phases — failure modes a short benchmark task is too brief to exercise.
• Primary-doc-backed: LangChain built IssueBench, a 15-task benchmark across three domains and 15 named failure categories, specifically because judging whether a trace-analysis tool works in production required a purpose-built eval, not a generic agent benchmark.
• Primary-doc-backed: Anthropic's own infrastructure-noise study measured a 6-percentage-point Terminal-Bench 2.0 swing (p < 0.01) from resource configuration alone, holding the agent fixed, plus a smaller 1.54-point swing on SWE-bench from RAM allocation — direct evidence that a benchmark score is not independent of the infrastructure it ran on.
• Editorial inference: a benchmark score is a single-round, fixed-distribution, fixed-infrastructure measurement; production reliability is a claim about repeated re-optimization, long-horizon behavior, and the infrastructure actually deployed, and only the team's own continual evaluation can measure that gap.

• Don't cite a benchmark number as a permanent property. If an agent or harness was optimized once against a benchmark, ask whether that gain has been re-checked after any subsequent re-optimization — a regressed or plateaued gain looks identical to a real one on a single before/after comparison.
• Build regression checks into your own optimization loop. When you tune a harness, prompt, or fine-tune against your eval set, re-run the previous eval set alongside the new one, the way RELAI-VCL's regression control caught what GEPA's optimization missed.
• Mine your own production traces, not just the public benchmark. LangChain's 13.7% harness lift came from trace-mining, not from re-running a published benchmark; a generic benchmark can't see the failure categories specific to your agent and your users.
• Build a benchmark for your specific failure categories once you know what they are. Follow IssueBench's pattern — name the failure categories you actually see (not generic ones), and test against multiple domains if your agent operates in more than one.
• Test long-horizon sessions, not just single tasks. Audit at least one long real session (hundreds of turns, not one) for compounding failures — deferred fixes, unverified assumptions, confidently wrong claims — the way the 241-turn Claude session write-up did, since a short benchmark task cannot produce this failure mode by construction.
• Control infrastructure before comparing scores. When comparing two agents, two models, or your own before/after, fix the resource configuration (CPU, RAM, kill thresholds) across the comparison, or treat a difference smaller than roughly 6 points as potentially infrastructure noise rather than a capability gap. Calibrate your own eval harness to about 3x resource headroom, per Anthropic's recommendation, so scores stop moving with compute budget.

• Treating a benchmark score as permanent: citing a pass rate from one optimization round as if it will hold after the next re-optimization, without re-checking.
• Optimizing without regression control: tuning against new tasks without also re-running the eval set the previous gain was measured on, so a regression (like GEPA's) goes unnoticed.
• Never mining your own traces: relying only on public benchmark numbers instead of mining production traces for the failure categories specific to your deployment.
• Testing only short tasks: validating an agent exclusively on benchmark-length tasks and never auditing a long real session, where compounding failures actually show up.
• Generic evals for a specific problem: using a general-purpose agent benchmark to judge a narrow tool (like a trace-analysis system) instead of building a targeted eval the way IssueBench does.
• Comparing scores across uncontrolled infrastructure: reading a leaderboard or before/after gap as pure capability difference without checking whether resource configuration differed enough to explain it on its own.

See agent evaluation for the broader problem of grading agent trajectories, agent benchmarks for how fixed-task benchmarks are constructed and where they fall short, agent tracing for the trace-mining half of this loop, and can you trust an LLM-as-judge score? for the adjacent problem of whether the grader itself is reliable.