LLM Digest
Subscribe

Agent Know-How

Agent engineering · know-how

Agents reach the outside world through fragile, ad-hoc integrations

🧱 Obstacle·tool-use·active·26 entries·27 sources·updated 2026-10-29

An agent is only as useful as the tools it can call, but every integration has historically been bespoke: hand-written wrappers around REST APIs, brittle schemas the model misuses, and no shared way to discover or authorize tools. Connecting an agent to real systems is where much of the engineering goes, and it breaks in production in ways the model never sees.

Latest updates
State of the art

The wire is standardizing on a protocol layer. MCP describes, discovers, and calls tools the same way for every agent, so the per-app glue question is mostly settled. Enterprises reach it two ways: purpose-built agent-native endpoints, or (more often) thin overlays and generated MCP surfaces in front of services that already run.

Discovery is now a retrieval problem. Once an agent can reach dozens of connectors, loading every schema burns context and degrades tool choice. Harnesses now search the tool catalog by default and keep intermediate results out of the model's context. Vendor numbers show this lifts accuracy, not only token budgets. Tool definitions (names, typed constraints, example inputs, result size) are treated as a design discipline.

Calling behavior is the open problem. Agents that pass clean tool suites degrade when tools time out or return bad data, and enabling JSON-schema output alongside tool calling can stop open-weight models from calling tools at all. The response is to tune or train the harness around the tools: search over harness configurations with frozen weights, or record harness rollouts as RL data.

Governance is moving earlier in the call. Beyond who may connect, teams now verify a proposed call before it runs and write policy over sequences of calls, since per-request checks miss concurrent abuse. Temporal policies need stateful tracking and give up some formal guarantees.

Production deployments keep the model's job narrow: orchestration and language, with specialized components, approvals, and isolated execution doing the rest.

Why it matters for platform engineers

Tool integration is the part of an agent that looks like ordinary distributed systems: auth, rate limits, retries, multi-tenancy, sandboxing. That is where most production incidents live, not in the model.

A protocol like MCP reduces N×M custom connectors to one interface, but it makes authorization and blast radius central: every exposed tool is a new permission and a new attack surface (see prompt injection and agent sandboxing).

The build-vs-buy decision is now "adopt the protocol and govern the connectors", plus testing how the agent behaves when a tool fails, rather than "write another API wrapper".

Developments 26

Exposing existing systems as agent-callable tools

4 entries · latest Sep 24, 2026

Teams either build agent-native endpoints or wrap what already runs, via thin overlays, generated MCP surfaces, or per-app capabilities; wrapping is winning in the enterprise.

Morgan Stanley generates MCP and A2A surfaces from its architecture-as-code model

Morgan Stanley models its API program with Architecture as Code (the CALM framework) and generates Model Context Protocol and Agent-to-Agent (A2A) integration surfaces from that description, with governance enforced through deployment gates.

Agent-callable tooling becomes a byproduct of API governance already in place, not a separate integration project with its own servers to hand-write.

Also relevant to Model Context Protocol: a standard interface for agent tools

Lovable exposes each published app as an MCP capability alongside its human UI

Lovable is branching into MCP-powered "capabilities", which CTO Fabian Hedin defines as a part of an application an agent can call directly without a human opening the app. A published app gets two interfaces: the human UI and an MCP endpoint any MCP client can call.

A connector gateway keeps credentials server-side and encrypted; generated app code gets a short-lived key scoped to one user instead of a secret. The protocol argument now reaches the individual SaaS app, not only infrastructure vendors.

Also relevant to Model Context Protocol: a standard interface for agent tools

Agentic overlays wrap existing REST services as MCP tools without rebuilding them

AWS and co-authors describe agentic overlays: thin wrapper layers in front of existing REST services that expose them as MCP-compatible tools and let them take part in Agent-to-Agent (A2A) interactions, without changing the underlying system.

This is the brownfield answer to "rebuild agent-native". It trades the cleanliness of purpose-built endpoints for reuse of what already runs in production.

Show 1 earlier entry

Agents need agent-native infrastructure, not human-oriented APIs with a wrapper

A messaging-infrastructure vendor argues that agents need purpose-built, agent-native endpoints, not SMS-style REST APIs designed for human developers and wrapped after the fact. Its path runs from plain REST APIs to an MCP surface.

It states the greenfield position: design the capability for an agent caller from the start. Most enterprises cannot rebuild their estate this way, which is why brownfield overlays appear alongside it.

Tool discovery and definition quality at scale

6 entries · latest Oct 27, 2026

Listing every schema in the prompt burns context and hurts tool choice, so harnesses now search the tool catalog and teams treat tool definitions as a design discipline.

Serving the RFC corpus through a skill or MCP server cuts reading tokens about 41x

RFC AI Tooling exposes the IETF RFC corpus to agents two ways, as a skill and as an MCP server, both backed by the same stdlib-only Python core. Documents are fetched from the RFC Editor on demand, and obsoleted specs are flagged so agents stop citing dead ones. The author reports a median 41x saving (tokens approximated as bytes divided by 4; single-repo, self-measured).

Takeaway: a thin find-read-flag interface over a large reference corpus beats pasting documents into context. A skill and an MCP server can share one core instead of two implementations.

Agents that query for skills mid-task beat static retrieval, but under-invoke and add latency

SkillBrowser targets Selective Skill Exposure: loading procedural skill content only when needed. Progressive disclosure lists metadata for the whole library and pollutes context. Static retrieval from the initial task query misses needs that emerge mid-reasoning.

A meta-skill has the agent name its current capability need, issue a targeted query, and load matching skills on demand. Across six SRA-Bench subsets and five LLMs (paper-reported), it improves aggregate accuracy over static retrieval and LLM-based selection, and limits accuracy loss on instances the model already solves.

The authors report model- and task-dependent gains, under-invocation of useful skills, and added latency. Large skill libraries need a retrieval step the agent controls, not a catalog dumped into the prompt.

Codex adds an MCP discovery grace period and a hook to rewrite tool results

Codex 0.151.0 adds a configurable grace period for discovering tools from optional MCP servers, and lets extensions inspect or replace an MCP tool's result before it reaches the model.

The result hook is a client-side filter on the tool-call path: a place to trim oversized payloads or redact fields without changing the server.

Also relevant to Model Context Protocol: a standard interface for agent tools

Show 3 earlier entries

Anthropic's tool search, programmatic calling, and input examples cut tokens and raise accuracy

Three features on the Claude Developer Platform (vendor-reported numbers):

  • Tool Search Tool: tools marked defer_loading: true are searched (regex or BM25) instead of loaded. A 50+ MCP-tool prompt drops from about 72K to about 500 tokens at rest (~3K per query); tool-heavy accuracy rises 49% to 74% on Opus 4 and 79.5% to 88.1% on Opus 4.5.
  • Programmatic Tool Calling: Claude orchestrates tools in sandboxed Python, so intermediate results stay out of its context. Tokens fell 43,588 to 27,297 on a research task; GAIA rose 46.5% to 51.2%.
  • input_examples: shows parameter conventions JSON Schema can't express; complex-parameter accuracy rose 72% to 90%.

On-demand discovery is a correctness fix, not only a context-budget one.

Also relevant to Model Context Protocol: a standard interface for agent tools

Bloated schemas and vague names are the main MCP tool-design failures; lazy loading halves context

AWS's MCP tool-design guide names two failure modes:

  • Bloated context: every tool schema loads on every call, used or not.
  • Confusion: vague parameter names and oversized result payloads make the model pick the wrong tool or call the right one wrong.

Its fix progression runs from the raw API exposed as-is, through richer descriptions and Literal-typed constraints, to lazy-loaded taxonomies behind a discovery tool. The leanest design cut per-turn context use from 4% to 2%. The guide cites Anthropic's lazy-loading work at up to 85% token reduction and suggests capping a tool at roughly eight parameters.

Also relevant to Model Context Protocol: a standard interface for agent tools

Making the calling loop reliable

8 entries · latest Oct 29, 2026

A standard wire does not make calling behavior reliable; flaky tools, schema-constraint interference, and connection failures are measurable failure modes, and the harness itself is now tuned or trained to fix them.

A successful tool call is not progress: an agent reran one SQL query 254 times

Funnel A/B-tested GLM-5.2 against Claude Opus 4.8 on its internal data agent (self-reported). GLM answered each model call faster (7.8s vs 18.1s) but made 66% more tool calls (19 vs 11) and finished tasks 14% slower end to end. Twice it reran the same successful query 113 and 254 times, because the tool returned only a 20-row preview and the model never recognized the limit.

Funnel traced it to the harness: ambiguous tool contracts, saved instructions naming tables that did not exist, and no repeated-call guard. For builders: label previews versus complete results, detect repeated equivalent calls, and cap execution budgets before swapping models.

Also relevant to Agent token costs are unpredictable and easily run away

An agent that can quote the tool instruction still skips the tool: access is not invocation

A developer exposed an ask_user MCP tool (speak a question, listen for the answer) to Claude Code. The agent used it in one project and printed the question on screen in the next, same machine, same day, even though it could quote the instruction back word for word.

This is a single practitioner report, not a controlled study. It shows that registering a tool and instructing its use does not guarantee invocation.

For builders: log tool-call rates per project or prompt context, and test that a tool is actually called, not just available.

A training-free validate-and-repair layer recovers tool calls broken by interface drift

Tool descriptions, argument schemas, and the candidate tool list often change after deployment while tool semantics stay fixed. A paper calls this interface drift and tests SchemaShield-Lite: schema validation, a candidate-set retry when the tool name is missing, and one-shot constrained repair, with no retraining.

On Qwen3.5-9B (paper-reported), BFCL-200 accuracy rises from 0.580 to 0.805 and DICE-300 from 0.717 to 0.813. It recovers 33 of 45 and 19 of 22 drift-induced failures with no observed repair harms. Naive retry does not explain the gain. Distractor tools in the candidate set cause most of the damage.

For builders: validate every call against the live schema and keep the candidate list stable.

Show 5 earlier entries

Claude Code makes MCP startup hangs and mid-session disconnects visible and boundable

Two Claude Code releases harden the MCP client's connection lifecycle:

  • v2.1.273 notifies the user when an MCP server disconnects mid-session and automatic reconnection gives up, pointing at /mcp.
  • v2.1.274 adds CLAUDE_CODE_MCP_STARTUP_WAIT_MS to bound how long the first non-interactive turn waits for MCP servers to connect (0 = don't wait).

Both were silent failures before: a hung startup and a dropped session nobody notices. Headless runs can now set an explicit startup budget.

Also relevant to Model Context Protocol: a standard interface for agent tools

StarHarness evolves the harness, tools included, for 20-35 point gains with frozen weights

StarHarness treats the whole harness as a search space while keeping model weights fixed: prompt and task framing, tool interfaces, skills, MCP-backed providers, subagent structure, and loop configuration. It stratifies tasks by how the default harness fails, separates proposal tasks from a hidden selection set, and holds out a third set to test generalization.

On ITBench SRE, EnterpriseOps-Gym ITSM, and AutomationBench Finance, the evolved harness beats the default by 20-35 percentage points after 4-12 accepted changes, and gains transfer across GPT and Qwen without re-evolving.

Where OpenForgeRL trains the model to fit the harness, this tunes the harness and its tool surface instead.

OpenForgeRL records harness model calls as RL data to train harness-native agents

Harnesses such as Claude Code, Codex, and OpenClaw are stateful, multi-process systems that open SFT/RL stacks cannot express, so training a harness-native agent end to end was out of reach. OpenForgeRL puts a lightweight proxy in front of the harness's model calls and records them as RL training data (for example, for veRL), with a Kubernetes orchestrator running each rollout in its own remote container.

It is validated on tool/harness agents and on GUI and browser-use agents, beating open baselines of similar size on nearly every benchmark tested (ClawEval, QwenClawBench, OSWorld-Verified, Online-Mind2Web, WebVoyager).

JSON-schema output constraints suppress tool calling in open-weight models

The "Constraint Tax" study reports a reproducible effect first seen in a production agent: with tool calling and JSON-schema constraints enabled together, several open-weight models stop invoking tools while still producing valid structured output.

The two core agent capabilities interfere. Forcing a clean output contract can quietly stop the agent from calling the tool it needed, so test the combination you deploy, not each feature alone.

Agents that pass clean tool suites degrade sharply when tools time out or return bad data

"Beyond Function Calling" introduces ToolBench-X, a benchmark for tool-environment unreliability: tools that time out, error, or return malformed or inconsistent results. Agents that look competent on clean, stable tool suites degrade sharply once the environment misbehaves.

A passing schema test is no evidence the agent recovers when the tool itself fails. Test tool-failure handling the way you would test any client of a flaky dependency.

Checking and governing tool calls before they run

4 entries · latest Aug 17, 2026

Controls are moving from scoping what a tool can touch to checking each call before execution and writing policy over sequences of calls, at the cost of stateful tracking.

AWS Dogwood extends Cedar so policies can reason over an agent's tool-call history

AWS open-sourced Dogwood (Apache 2.0), which adds temporal operators to Cedar: formerly, count_within, count_distinct_within, sum_within. Rules can read prior tool calls instead of judging one request alone. It is supported in AgentCore Policy, but the reference interpreter is not production-ready.

The motivating case: a $5,000 transfer cap that checks settled responses is defeated by concurrency, because three simultaneous $2,000 requests all pass. The rule must count requests within a time window.

The cost is real: temporal evaluation needs stateful event tracking and gives up Cedar's automated formal-reasoning guarantees.

A static verifier checks OpenCode tool calls against safety properties before they run

A Show HN plugin brings the formal-verification work in "Guardians of the Agents" (Erik Meijer; implemented by Nada Amin) to OpenCode. It hooks tool.execute.before to intercept candidate bash, read, edit, and write calls and verifies them statically before execution.

It is a pre-execution check, complementary to sandboxing and authorization, which limit what happens after a call runs. See agent sandboxing.

A protocol layer of versioned virtual tools with taint tracking to stop exfiltration

Jake Mannix proposes an intermediate protocol layer that turns raw APIs into versioned, encapsulated "virtual tools", with interface mapping, dynamic schema projection, and runtime taint tracking to catch data-exfiltration risk at the tool boundary.

This is one engineering leader's architecture, not a benchmarked result. It targets ungoverned tool sprawl through versioning and data-flow tracking rather than schema hygiene alone.

Show 1 earlier entry

Running tool-calling agents in production

4 entries · latest Sep 18, 2026

Cloud runtimes now host the tool loop, and production deployments narrow the model to orchestration while specialized components, approvals, and isolation carry the work.

DoorDash cleans stale feature flags with MCP-fed agents at $4.79 per cleanup

DoorDash built a multi-agent system to clean up stale flags across more than 60,000 flags and 623 repositories. It pulls live experimentation data through MCP, gates each cleanup on engineer approval, and runs parallel agents in isolated Git worktrees with automated validation.

In an evaluation of 50 flags, 45 produced usable pull requests at an average of 13.8 minutes and $4.79 per cleanup (company-reported). MCP here is the access layer into internal engineering systems, not a customer-facing feature.

Also relevant to Model Context Protocol: a standard interface for agent tools

DoorDash's shopping assistant keeps the LLM to orchestration and language

Ask DoorDash splits work across specialized agents, MCP-based tooling, and a separate intelligence layer with persistent consumer memory and live backend data, rather than one model deciding everything. DoorDash reports up to 24% higher checkout conversion and 17% larger baskets from memory-backed sessions (company-reported).

The pattern narrows what the model owns: deterministic and specialized components carry the task, and the LLM routes and talks.

Azure Functions folds the tool-calling loop into a serverless agents runtime

Azure Functions shipped a serverless agents runtime in public preview at Build 2026. An agent is an .agent.md file with YAML triggers, MCP server access, 1,400+ connectors, and sandboxed execution. The Functions team told InfoQ the runtime adds no cold-start overhead and no billing premium beyond standard Flex Consumption (vendor-reported).

MCP access becomes a default runtime feature rather than something each team wires in, and the cloud platform takes over hosting the tool loop.

Also relevant to Model Context Protocol: a standard interface for agent tools

Show 1 earlier entry

Hosting browsers for agents is a distributed-systems problem, not a model problem

Paul Klein's talk on cloud-hosted browser infrastructure for agents covers bursty, stateful multi-tenancy and securing Chromium against remote code execution with Firecracker microVMs. MCP is the layer that turns complex websites into tools an agent can call.

The model's tool-calling skill does nothing for these failures. Capacity, isolation, and sandbox escape are ordinary infra work that the tool layer inherits.

From related topics
  1. Model Context Protocol: a standard interface for agent toolsLangChain's MCP client maps elicitation to a LangGraph interrupt and caches tool lists
  2. Model Context Protocol: a standard interface for agent toolsClaude Agent SDK for Python supports MCP 2.x alongside 1.x for in-process servers
  3. Model Context Protocol: a standard interface for agent toolsAzure DevOps Remote MCP Server reaches GA, but Claude, ChatGPT, and Cursor can't connect
  4. Model Context Protocol: a standard interface for agent toolsDevelopers split on whether stateless MCP is just REST again
  5. Model Context Protocol: a standard interface for agent toolsCloudflare turns on WebMCP for unmodified pages with a dashboard switch
  6. Model Context Protocol: a standard interface for agent toolsAzure API Management's AI Gateway tier governs models and MCP servers in one control plane
  7. Model Context Protocol: a standard interface for agent toolsAWS bridges cloud-hosted agents to MCP servers on a user's laptop with no open ports
  8. Model Context Protocol: a standard interface for agent toolsThe 2026-07-28 spec makes MCP stateless, adds governed extensions, hardens auth
  9. Model Context Protocol: a standard interface for agent toolsSecuring MCP in production needs four control layers, not just a gateway
  10. Model Context Protocol: a standard interface for agent toolsEuclid-MCP delegates logical inference to SWI-Prolog and beats LLMs on rule enforcement
  11. Model Context Protocol: a standard interface for agent toolsMCP's Enterprise-Managed Authorization extension reaches stable
  12. Model Context Protocol: a standard interface for agent toolsWebMCP enters Chrome origin trials, letting sites expose functions and forms to agents
Evidence · 27 sources
All sources behind this page