Agent Engineering Wiki

Benchmark what the agent did, not just its answer: rubric-style suites score whether the right tools were called and the task was actually completed, and structural benchmarks probe specific failure axes (e.g. DPBench on the determinants of multi-agent coordination).

Measure capability on your own tooling and out of distribution: Hugging Face's "is it agentic enough" workbench benchmarks open models against the caller's actual tools, and "Running the Gauntlet" shows agents that top familiar leaderboards degrade sharply in unfamiliar environments — so a high public score is weak evidence for your workload. Reusable eval workbenches (olmo-eval) package this into the model/agent development loop so benchmarking is a standing harness, not a one-off.

The harness is part of what you benchmark: a cross-harness study reports a deliberately simple agent loop reaching SOTA across 21 models on SWE-pro and Terminal-Bench-style suites, evidence that elaborate scaffolding often adds cost and variance without adding capability — so the benchmark should hold the harness fixed and let it earn its complexity. Vendors are running this in-house: GitHub's evaluation of its Copilot agentic harness across 20+ models and many tasks scores results *and* token efficiency together, treating the scaffold as a benchmark variable and elevating cost-per-solved-task to a first-class metric alongside accuracy.

Mined from real sessions: rather than synthetic tasks, the newest suites are mined from real sessions — EnterpriseClawBench builds enterprise-agent tasks from actual workplace sessions where an agent reads heterogeneous files, calls tools, and has to deliver a business artifact, so the benchmark inherits the messiness of production instead of approximating it.

Reproducibility is the flip side of trusting a benchmark: because agent runs touch the network, filesystem, and shifting tool versions, a score only means something if the environment is fixed — Proctor packages coding-agent benchmarks as signed, isolated bundles so a run can be reproduced (and a leaderboard claim audited) rather than taken on faith.

Adversarial tool environments: rather than assuming tools behave, "Beyond Function Calling" scores agents when tools time out, error, or return malformed results, exposing agents that pass clean tool suites but cannot recover when the environment misbehaves — the benchmark targets the *failure recovery* path, not the happy path.

Held-out, hard-to-memorize tasks: practitioners are reaching for novel environments a model can't have trained on (a Sherlock Holmes deduction board game run as an LLM-agent eval) precisely because familiar leaderboards leak into training. Both this and the adversarial-tool-environment axis answer a gap practitioners keep voicing — public threads asking "what benchmarks actually compare agent *harnesses*" (beyond Terminal-Bench) — that the standard model leaderboards don't fill.

Subsystem-specific benchmarks isolate one capability instead of scoring end-to-end task success: a suite for the failure modes of agent memory (forgetting, stale recall, poisoned entries) and OpenRCA 2.0's shift from outcome labels to causal process supervision for root-cause analysis both grade an inner subsystem — the memory layer, the reasoning trajectory — so a regression can be localized to the part that broke rather than inferred from a fallen aggregate score. A microservice-failure-diagnosis benchmark (AgentOps) extends the same process-over-outcome grading to ops agents, scoring the diagnosis path over multimodal trace data and pulling benchmarking toward observability.

Eval transparency is improving too, on the meta side: Hugging Face now surfaces community "Every Eval Ever" results directly on model pages, making the spread of scores visible rather than relying on a single headline number.

The domain-specific and long-horizon fronts are both advancing: ScarfBench narrows to a single high-stakes enterprise task (migrating Java frameworks) rather than a generic coding benchmark, following the "mined from real work" pattern EnterpriseClawBench set; and Emergence World is built specifically to grade long-horizon autonomy — sustained multi-step operation rather than a single bounded task — the harder distribution-shift edge the "familiar leaderboards degrade out of distribution" finding already flags.

Benchmark upkeep is being automated, addressing the standing trade-off that a hand-built benchmark is real work to author and maintain: Reap automates curation of coding-agent benchmark tasks rather than requiring a team to hand-pick and refresh them. A new environment-readiness angle also appears: AeroScore scores how well existing documentation portals support AI agents in the first place, evaluating the environment an agent has to operate in rather than the agent itself — a precondition check that sits upstream of any task benchmark. On the subsystem-specific front, TestEvo-Bench adds an executable, live benchmark for test-and-code co-evolution, isolating whether an agent keeps tests in sync with the code it changes. And a new capability frontier opens on program understanding: MirrorCode benchmarks agents rebuilding entire programs from behavior alone (black-box reconstruction), pushing past "modify existing code" into "reconstruct it from how it behaves." The domain-narrow list keeps growing: GameEngineBench scores coding agents against real C++ game-engine runtime environments, extending "mined from real work, one domain at a time" (alongside ScarfBench's Java migrations) into a runtime with real-time simulation, physics, and rendering constraints a generic coding benchmark doesn't exercise.

The domain-narrow benchmark list widens to game engines: GameEngineBench scores coding agents against real C++ runtime environments (simulation, physics, rendering), the same "mined from a real, high-stakes domain" pattern as ScarfBench's Java migrations.

Benchmark noise is now quantified rather than assumed: run-to-run standard deviation for a single model on a coding task measured 7.5% — bigger than the gap between the best- and worst-ranked models — and dropping a handful of tasks from a ~100-task set was enough to flip the ranking, evidence that a small task set's variance can dominate the signal it's meant to measure.

A fixed benchmark is reproducible and cheap to re-run, but it's a static target: agents over-fit to it, it goes stale as tools change, and "passing" can mean "memorized the distribution."

Building a benchmark on your own tooling is more predictive but is real work to author and maintain, and small task sets have high variance — measured, not just suspected: one practitioner found a model's own run-to-run standard deviation (7.5% on a coding task) exceeded the best-to-worst-model gap, and swapping a few tasks out of a ~100-task set flipped which model ranked first. Two models can also both look "cheaper" and "more expensive" than each other depending on which tasks the comparison uses — so a single leaderboard number is a claim about that task set, not a general fact about the model.

Best as a regression gate (catch known failures) — complement with LLM-as-judge on live traces for the open-ended cases a fixed suite can't enumerate.

Agent benchmarks are the CI gate of the agent stack: a fixed suite you run on every prompt, model, or tool change to catch regressions before users do.

The leverage is building it from *your* environment and tools, because public leaderboards systematically over-state how an agent will do on your workload — and budgeting the upkeep, since a benchmark is only useful while it still resembles production.