Agent Know-How

A protocol-validity analysis argues that many agent benchmarks conflate task difficulty with scaffolding and protocol artifacts. Two agents can score differently because of how their harness interacts with the benchmark's protocol, not because one is more capable.

This turns "the harness is part of what you benchmark" into a challenge to the benchmark's validity as a measuring instrument, not only its reproducibility.

OpenBench scores different coding-agent harnesses against each other on the same tasks.

It is a dedicated answer to the practitioner question of which harness works best, instead of repurposing a model-comparison benchmark.

Favur is a 14-agent harness (planner, architect, coder, tester, reviewer, builder) coordinated by code, not an LLM. Favur Evals scores its runs across models from the same standardized statement of work, using composite engineering subjects computed from each run's own artifacts: lint, test results, tool telemetry.

Every run is captured end to end and can be replayed. Reproducibility becomes a feature of the benchmark rather than a property to demand of it.

Dan Luu's notes on agentic coding measure benchmark variance directly: one model's run-to-run standard deviation (7.5% on a coding task) exceeded the best-to-worst-model gap, and swapping a few tasks out of a ~100-task set flipped which model ranked first. Two models can each look cheaper or more expensive than the other depending on the task set.

A leaderboard number is a claim about that task set, not a general fact about the model. Report variance across repeated runs.

GitHub evaluated its Copilot agentic harness across more than 20 models and multiple benchmarks, reporting results and token efficiency together.

Treating the scaffold as a benchmark variable and cost per solved task as a first-class metric is the comparison builders need when choosing a model inside a fixed harness. The results are GitHub's own.

Hugging Face now surfaces community "Every Eval Ever" results directly on model pages.

Readers see the spread of scores across many evaluations instead of relying on one headline number a vendor chose to publish.

An Ask HN thread asks which benchmarks show how different models do in different harnesses, beyond Terminal-Bench, which the poster says doesn't match their experience.

Model leaderboards don't answer the question practitioners actually face: which model-plus-harness combination works on their tasks.

Agent runs touch the network, filesystem, and shifting tool versions, so a score only means something if the environment is fixed. Proctor packages coding-agent benchmarks as signed isolation bundles.

A run can then be reproduced, and a leaderboard claim audited, rather than taken on faith.

Strands Labs' benchmark-harnesses repo reports a deliberately simple agent loop reaching state-of-the-art results across 21 models on SWE-bench Pro, Terminal-Bench 2, and SWE-bench Verified.

Elaborate scaffolding often adds cost and variance without adding capability. Hold the harness fixed when benchmarking and make extra complexity earn its place.

AeroScore scores how well documentation portals support AI agents, using the open AFDocs checklist (llms.txt presence, robots.txt, and more) plus a PDF-heaviness extension. None of the six aerospace portals in its first batch scored as sufficiently prepared.

It evaluates the environment an agent has to operate in, not the agent itself: a precondition check that sits upstream of any task benchmark.

A practitioner runs a Sherlock Holmes deduction board game as an LLM-agent eval to measure how good a detective an AI is.

Novel environments a model can't have trained on are one answer to leaderboards leaking into training data.

EnterpriseClawBench mines 852 reproducible tasks from an archive of real enterprise agent sessions where agents read heterogeneous files, call tools, and deliver business artifacts. Each task pairs recovered fixtures with rewritten prompts, role classes, hard rules, and semantic rubrics.

The benchmark inherits the messiness of production instead of approximating it. The catch: the sessions contain internal content, so the data is not released.

"Running the Gauntlet" introduces GauntletBench, a web-based benchmark for agent generalization in challenging scenarios. It argues current benchmarks are built on popular applications with simple tasks, so modern agents saturate them without exposing their limits.

Agents that score well on familiar leaderboards degrade sharply outside familiar environments. A high public score is weak evidence for how an agent will do on your workload.

Rubric is an open-source tool that scores an agent's actions: whether it called the right tools and actually completed the task, not only whether its final answer reads correctly.

Grading the trajectory turns an agent eval into an integration test. A correct-sounding answer reached through the wrong tool calls fails.

AllenAI's olmo-eval is an evaluation workbench built into the model development loop, so benchmarking runs as a standing harness rather than a one-off report.

A reusable harness is what makes a benchmark a regression gate: the same suite runs on every change instead of being rebuilt per release.

Hugging Face's "Is it agentic enough?" shows how to benchmark open models against your own tooling instead of a public suite's tools.

Which model works best depends on the tools it has to call. Testing on your actual tool definitions predicts production behavior better than a leaderboard built on someone else's.

SWE-Touch stress-tests coding agents in a shared workspace with validated Counter-Edits: plausible user edits to task-relevant code that conflict with completing the task, made while the agent is still working.

Repository benchmarks usually assume an uninterrupted solo run. Real pairing with a coding agent doesn't work that way.

ExtractBench evaluates schema-guided extraction: given a document and a user-defined schema, produce correct output with source evidence. It spans 4,869 pages across 370 enterprise documents, 8 business domains, and 67 document types, and is the first to score value accuracy, record completeness, grounding, and measured cost together.

A bare accuracy number hides whether an extraction can be traced to its source and what it cost.

LangChain's ReviewBench evaluates code-review agents against real PR feedback from trusted human reviewers instead of a synthetic rubric.

Ground truth mined from actual review decisions is harder to game and closer to what a team will accept.

Anthropic and Andon Labs built Drone-Bench to test whether AI models can autonomously fly a drone to locate and follow a person.

It extends "exercise real tool use" from software into embodied control, where the tool is a physical actuator with real-world latency and failure modes rather than an API.

HalluTruthQA has 2,400 expert-curated Arabic QA examples across four domains (Islamic knowledge, history, science, geography). Each answer has a verified reference and six verification candidates; hallucinated answers carry character-level error spans and human-written explanations.

Zero-shot, no open-source model tops every sub-task. Grading is moving past English-centric, response-level hallucination labels toward non-English, finer-grained ones.

Stripe's 11-environment benchmark scores agents on real Stripe integrations: checkout migration, billing API work, and full-stack browser checkout. The best runs needed roughly 63 interaction turns.

The two leading models (92% vs 73%) failed the identical validation step rather than differing on raw coding ability. Validation, not code generation, is where integration agents break.

Imaging-101 scores coding agents on 57 expert-verified computational-imaging tasks across six scientific domains, each grounded in a peer-reviewed paper and canonicalized into a four-stage pipeline. Three tracks cover planning, unit tests, and end-to-end reconstruction.

Failures are domain-specific, such as physical-convention handling and pipeline integration, beyond generic coding skill.

GameEngineBench evaluates coding agents on scoped C++ implementation tasks inside Unreal Engine, where code must work in a stateful, interactive system with real-time simulation, rendering, and physics.

Those runtime constraints, relevant to robotics, healthcare, and manufacturing 3D applications too, aren't exercised by a generic coding benchmark.

TestEvo-Bench is an executable, live benchmark of test-and-code co-evolution tasks mined from real repositories. Unlike benchmarks that isolate the test from the code change or rely on static metadata, it verifies that generated tests run and are semantically tied to the change.

It isolates a habit coding agents need in production: keeping the test suite in sync with what they modify.

MirrorCode is a long-horizon coding benchmark where agents reimplement entire software projects by replicating an existing program's functionality. It targets the gap between short-task benchmarks and one-off demos (such as an AI-built C compiler) that are hard to compare because they involve human guidance and aren't repeated across models.

It moves the frontier from "modify existing code" to "reconstruct it from how it behaves."

IBM Research's ScarfBench narrows to one high-stakes enterprise task: migrating Java applications between frameworks.

A domain-specific suite tells a team doing that work far more than a generic coding score does.

IssueTrojanBench tests whether a coding agent executes a malicious instruction smuggled inside an otherwise ordinary GitHub issue.

It moves adversarial testing from malformed tool output to malicious task input, the channel a coding agent trusts most. See prompt injection.

ActionRail's benchmark tests whether an agent executes corrupted-but-plausible business data, such as an altered payment account or a fake refund address, buried in an otherwise legitimate document.

Across 8 models from 4 providers on 10 consequential workflows, cost-optimized models failed 48.3-63.3% of the time versus 1.7-21.7% for frontier models. A guard layer blocked all 480 protected attack cases with zero false positives. This failure mode needs a dedicated defense, not just a stronger model. The guard layer and results are ActionRail's own.

"Beyond Function Calling" introduces ToolBench-X, which tests agents under recoverable reliability hazards: tools that time out, error, or return malformed results, across sequential, parallel, and mixed workflows with deterministic tools.

It exposes agents that pass clean tool suites but cannot recover when the environment misbehaves. The benchmark targets the failure-recovery path, not the happy path.

OmniaBench derives an application-oriented taxonomy from app stores, product docs, and web retrieval, spanning 1,431 tasks across 90 top-level domains with explicit state spaces.

Even frontier models clear only about half the suite, headroom that narrower coding and tool-use benchmarks don't surface.

Emergence AI built Emergence World specifically to evaluate long-horizon autonomy: sustained multi-step operation rather than a single bounded task.

Long horizons are the hardest distribution shift for an agent. A suite that only scores short tasks cannot tell you whether an agent stays on track over hours of operation.

Two datasets (AIOps2025 and RCA100) evaluate LLM agents on microservice failure diagnosis over multimodal observability data. Grading covers the reasoning process along three dimensions, starting with localization (where the fault is) and identification (what type), rather than only the final answer.

It brings process-over-outcome grading to ops agents and pulls benchmarking toward observability.

OpenRCA 2.0 argues that labeling only the root cause reduces RCA to pattern matching. Its PAVE protocol uses known fault-injection interventions to reconstruct the causal propagation path from root cause to symptom, and grades agents step by step.

Grading the reasoning trajectory localizes a regression to the step that broke, instead of inferring it from a fallen aggregate score.

An open-source benchmark targets the failure modes of agent memory, such as forgetting, stale recall, and poisoned entries, instead of end-to-end task success.

Testing the memory layer on its own tells you whether a regression came from memory or from the rest of the agent.

DPBench is a structural benchmark for multi-agent LLM coordination: instead of one end-to-end score, it probes which structural factors determine whether agents coordinate successfully.

Isolating one failure axis tells you why a multi-agent setup underperforms, which an aggregate task score cannot.