Agent Builder Foundations

Average success-rate improvement hides that skills cut both ways. Splitting outcomes into regressions (previously-passing tasks that now fail) and residual failures (tasks that never passed either way) shows the best skills mostly win by regressing less, not by solving more. Regressions cluster into three specific mechanisms, and the same three areas — procedural guidance, grounding, and verification — also explain most of what's left unsolved.

Treat a skill as a change to the agent's whole context, not a subroutine that only runs when invoked. A skill sitting in context can shift behavior on tasks that never call it, override how the agent reads its own inputs, and quietly turn off checks the agent would have run anyway. None of that shows up if you only measure "did the task pass," because a pass/fail count doesn't distinguish a task that was already broken from one your own change just broke.

The study runs agents with and without a candidate skill across nearly 6,000 tasks, two office-automation benchmarks, and three harness stacks, then buckets every outcome change into one of two categories: a regression (solved without the skill, failed with it) or a residual failure (failed either way). This decomposition is the whole point — it separates "the skill didn't help" from "the skill actively broke something that worked."

Three mechanisms explain most regressions:

• Skill description osmosis — the skill's presence in context changes agent behavior even on turns where the agent never invokes it. The text doesn't have to run to have an effect.
• Grounding displacement — the skill's prescribed procedure overrides how the agent interprets its actual inputs, so the agent follows the recipe instead of what's in front of it.
• Verification displacement — the procedure supplies its own sense of "done," which suppresses the output checks the agent would otherwise perform.

Looking at residual failures (tasks that still fail with the skill) turns up a matching imbalance rather than a different problem: existing skills over-invest in procedural guidance, the stage the study finds is least often the actual cause of failure, while under-supporting grounding and verification, the stages responsible for most of what's left unsolved.

Benchmark/result-backed: nearly 6,000 runs across two office-automation benchmarks and three harness stacks, with outcomes decomposed into regressions vs. residual failures rather than reported as a single success-rate delta. The paper reports the direction and mechanism of the effect (regressing-less beats gaining-more among top skills; three named regression causes) without publishing a specific percentage for how much of the improvement each mechanism explains — treat the mechanism finding as established and any percentage as unstated by the source.

• Score two numbers, not one. For every candidate skill, measure tasks newly solved and previously-passing tasks now failing separately — never collapse them into a single success-rate delta before shipping.
• Audit failures for the three named modes. When a task regresses, check whether the skill changed behavior on a turn it wasn't invoked on (osmosis), overrode input interpretation (grounding displacement), or suppressed an output check (verification displacement) before rewriting the procedure itself.
• Write grounding and verification steps as explicitly as the procedure. If a skill spells out steps but leaves "check your inputs" and "check your output" implicit, it's the shape most likely to displace exactly those checks.
• Re-test previously-passing tasks whenever a skill changes. A skill update that only gets evaluated against its target tasks will never surface a regression on tasks outside that set.

• Grading a skill by aggregate success-rate improvement, which lets regressions on previously-solved tasks hide behind gains on new ones.
• Assuming a skill only affects behavior when the agent actually invokes it, missing osmosis effects from the skill merely being present in context.
• Treating every regression as a procedure-writing problem and iterating on the steps, when the study finds grounding and verification gaps cause most of the remaining failures.
• Shipping a skill update without re-running the agent's previously-passing task set, so a new regression ships silently.

See tool use for the broader set of failure modes in connecting agents to real tools, and agent reliability for why fluent, confident-looking output doesn't imply the agent's checks are still running.