Agent Builder Foundations

No, not by itself. Three gaps separate the score from production behavior:

• Optimization rounds. On Terminal-Bench 2.0, one optimizer fell below the unoptimized baseline on new tasks and another plateaued; only a regression-controlled method kept improving.
• Horizon. A real 241-turn coding session surfaced compounding failures a short task cannot produce.
• Infrastructure. Holding the agent fixed, resource configuration alone moved Terminal-Bench 2.0 by 6 points, about the spread between top leaderboard models.

A benchmark answers "did it pass this set once, on this setup." Production asks whether it keeps working as tasks, optimization rounds, session length, and infrastructure change.

Think of a benchmark run as one frame of a video: accurate for that frame, silent about the next. Three forces move the video forward:

1. Re-optimization drift. Every new harness setting, prompt, or fine-tune is another optimization step. A gain measured once can shrink or turn negative after the next round meets tasks the last one never saw.
2. Horizon compounding. Benchmark tasks last minutes; production sessions run hundreds of turns. One wrong assumption is cheap until 50 more turns build on it with no checkpoint.
3. Infrastructure confounding. CPU, RAM, and kill thresholds move the score. A leaderboard that doesn't control them compares compute headroom as much as capability.

The response is the same for all three: measure your own agent across rounds, long sessions, and the infrastructure you deploy on, using your own traces.

A benchmark score runs an agent, optionally after one optimization step, against a fixed task set once and reports the aggregate pass rate. Three structural gaps separate that number from production reliability.

Optimization gains may not transfer. When a deployed agent is re-optimized as new failures surface, the previous round's gain meets tasks it never saw. In the Terminal-Bench 2.0 continual-learning study, only the method with regression checks inside its loop kept improving (76.4% vs. a 58.7% baseline); the others regressed or plateaued. All three could report a similar first-round number.

Short tasks can't produce long-horizon failures. The failures that matter at hundreds of turns are commitments: the agent adopts an unverified belief and builds on it fast. A wrong factual claim, a deferred fix, or an untested assumption only becomes expensive once later work depends on it. A single short task has neither the length to produce that nor the structure to catch it.

Scores include infrastructure. Resource limits decide which runs get killed or starved, so the score mixes capability with configuration. Anthropic measured a 6-point swing from resources alone, and past about 3x headroom extra resources still raised success while barely cutting infrastructure errors. The resource budget moves the score even after crashes are mostly gone, so a leaderboard gap of that size may be measuring compute, not capability.

The fix is your own continual evaluation. Mine production traces for the failure categories you actually see and build evals around them; LangChain's IssueBench names 15 categories instead of relying on generic pass/fail, and its trace-mining loop produced a 13.7% harness lift. Re-check every gain after the next optimization round, and hold infrastructure constant when comparing scores.

• Don't cite a benchmark number as permanent. Ask whether an optimization gain was re-checked after later re-optimization; a regressed gain looks identical to a real one in a single before/after.
• Put regression control in your optimization loop. Whenever you tune a harness, prompt, or fine-tune, re-run the previous eval set alongside the new one.
• Mine your own production traces. A public benchmark can't see the failure categories specific to your agent and your users.
• Build evals for the failure categories you can name. Follow IssueBench: specific categories, and multiple domains if your agent spans them.
• Audit at least one long real session. Read hundreds of turns for deferred fixes, unverified assumptions, and confidently wrong claims; a short task cannot produce them by construction.
• Control infrastructure before comparing scores. Fix CPU, RAM, and kill thresholds across a comparison, and treat gaps under about 6 points between uncontrolled setups as possible noise. Calibrate your harness to roughly 3x headroom with a hard kill threshold.

• Permanent scores: citing a one-round pass rate as if it survives the next re-optimization.
• No regression control: tuning on new tasks without re-running the old set, so a GEPA-style regression goes unnoticed.
• Benchmark-only signal: never mining production traces for the failure categories your deployment actually has.
• Short-task-only testing: never auditing a long session, where compounding failures show up.
• Generic evals for narrow tools: grading a trace-analysis system with a general agent benchmark instead of a targeted eval.
• Uncontrolled infrastructure: reading a leaderboard or before/after gap as capability without checking whether resource configuration explains it.

See agent evaluation, agent benchmarks for how fixed-task benchmarks are built, agent tracing for the trace-mining half of the loop, and can you trust an LLM-as-judge score? for whether the grader itself is reliable.