🧠 Agent Engineering Wiki

TL;DR

Pin down a fixed set of tasks with known good outcomes and run agents against them repeatedly. Unlike model benchmarks, agent benchmarks have to exercise *tool use and multi-step trajectories* — booking, querying, fixing, coordinating — so they double as integration tests for the whole agent, not just the model.

State of the art

Two themes dominate. First, benchmark what the agent did, not just its answer: rubric-style suites score whether the right tools were called and the task was actually completed, and structural benchmarks probe specific failure axes (e.g. DPBench on the determinants of multi-agent coordination). Second, measure capability on your own tooling and out of distribution: Hugging Face's "is it agentic enough" workbench benchmarks open models against the caller's actual tools, and "Running the Gauntlet" shows agents that top familiar leaderboards degrade sharply in unfamiliar environments — so a high public score is weak evidence for your workload. Reusable eval workbenches (olmo-eval) package this into the model/agent development loop so benchmarking is a standing harness, not a one-off.

What's new

Skepticism of leaderboard scores is now the default stance: results that agents collapse "beyond familiar environments," plus workbenches that benchmark models on *your* tools, push teams toward task suites grounded in their own environment rather than trusting a public number.

Trade-offs

A fixed benchmark is reproducible and cheap to re-run, but it's a static target: agents over-fit to it, it goes stale as tools change, and "passing" can mean "memorized the distribution." Building a benchmark on your own tooling is more predictive but is real work to author and maintain, and small task sets have high variance. Best as a regression gate (catch known failures) — complement with LLM-as-judge on live traces for the open-ended cases a fixed suite can't enumerate.

Why it matters for platform engineers

Agent benchmarks are the CI gate of the agent stack: a fixed suite you run on every prompt, model, or tool change to catch regressions before users do. The leverage is building it from *your* environment and tools, because public leaderboards systematically over-state how an agent will do on your workload — and budgeting the upkeep, since a benchmark is only useful while it still resembles production.

Addresses