Agent Know-How

Cloudflare put Traces into open beta, extending automatic tracing beyond Workers to security rules, transformations, cache, routing, and origin handling, all as OpenTelemetry spans. It accepts and forwards W3C traceparent headers, so edge spans attach to the trace your agent harness already started. Trace Rules sample selectively.

For builders, tool calls and model-gateway traffic that cross the edge no longer break the trace at the proxy. Pricing moves to ingestion and retention volume from December 1, so budget for span volume from agent loops before enabling it broadly.

Pond, a Show HN project, stores agent sessions from multiple machines in one S3 bucket you own, with no database service to run. Sessions are searchable and exposed to the agent as an MCP server.

Session history becomes a durable, portable record under the team's control rather than local files tied to one laptop or tool.

Cloudflare added agent spans to existing Workers traces: invoke_agent, chat/execute_tool, and tool_approval, keyed by agent name, agent ID, and conversation ID, so a session replays turn by turn.

The gotchas are concrete:

• Payload recording defaults differ by framework: off under Vercel's AI SDK, on under Flue. Those payloads often carry personal data or secrets.
• Traces are not lossless; payloads may be truncated, dropping the arguments a debug session needs.
• From October 1, 2026, every span is a billable event.

LangSmith's Trajectories renders a whole agent session as a conversational thread: user turns, tool calls, and subagent handoffs read top to bottom, instead of the nested span tree a generic OpenTelemetry viewer shows.

It trades completeness of the tree for the job that bottlenecks debugging long-running agents: scanning a session fast enough to spot where it went wrong.

InfoQ coverage of agent observability practice treats tool-call loops and runaway spend as the same triage step: both show up in the same session trace, and both need enough preserved execution context for post-incident debugging.

Cost dashboards and trace viewers are one diagnostic surface. See cost controls.

Amazon OpenSearch Service's MCP Apps return an interactive visualization alongside each tool call's text response. A local MCP server authenticates with AWS credentials, queries the same OpenSearch and Prometheus sources behind existing dashboards, and renders the result inline in the IDE.

One conversation moves from alert to log clustering, trace waterfalls, and service maps without opening a separate tool. MCP carries the observability payload itself.

Figma built alert-triage agents on a Panther SIEM foundation, querying 100+ data sources (AWS, Okta, GitHub, GCP, osquery) and scoped to the tools an on-call engineer uses. The team reports memory, not model choice or tool count, as the biggest quality lever.

Reported results: 70% faster resolution on complex alerts, 20% fewer on-call pages from re-tuned severity, and 100+ previously unknown vulnerabilities found. Agent-authored PRs default to draft and every fix needs human approval.

LangChain built an autonomous SRE agent for Kubernetes on Deep Agents. It requires human approval before applying a change, and every step, tool call, and decision is captured in LangSmith traces and covered by evals.

The trace is what a human reviews before the agent acts, not only what an engineer replays afterward.

ORCA-bench pairs an OpenTelemetry-instrumented microservice testbed (six days of metrics, logs, and traces) with 1,079 oncall RCA tasks, graded by an LLM judge that human SREs re-scored (κ=0.90). Across five frontier agents, the best accuracy is 25.3% on realistic-input tasks and 10.0% on hard ones. The weakest model invents an implausible root cause on 40% of reports.

The authors read this as a lower bound on the real-world gap. The reasoning may be there, but the end-to-end oncall pipeline is mostly unsolved.

LangSmith's Bring Your Own Cloud option is generally available on AWS, giving enterprise teams managed observability, evaluation, and deployment inside their own VPC.

It meets the "keep traces in our network" requirement from the vendor side: the Claude Apps Gateway answers it by self-hosting, BYOC by deploying the managed product into the customer's cloud.

AWS and Anthropic released the Claude Apps Gateway, a single stateless container an organization runs in front of Claude Code and Claude Desktop. It centralizes identity, policy, routing, and telemetry, relays per-request usage metrics to the team's own OpenTelemetry collector (CloudWatch, Prometheus), and enforces YAML-defined spend caps by org, group, or user.

Telemetry relay and cost policy live in one customer-owned layer instead of a vendor dashboard.

Schneider Electric built its enterprise LLMOps foundations on LangSmith to unify observability, evaluation, and deployment for AI products at scale (vendor case study).

It is an enterprise deployment of the trace as the shared substrate for ops and eval, not just a vendor pitch for it.

Latency, error rate, and token counts say nothing about whether an answer was grounded or useful. Grafana proposes running evaluations per interaction (grounded, request fulfilled, toxic, PII exposed, injected) and treating each result as an SLI. Checks can be an LLM judge or deterministic rules such as regexes.

Each behaviour gets its own objective instead of one "quality score". Example: 95% of conversations judged fulfilled over 30 days; the other 5% is the error budget. While budget remains, teams can change prompts or models. Once it burns, effort shifts to quality.

For builders, budget burn surfaces slow drift (a prompt, model, or retrieval change costing a fraction of a point per day) that error-rate alerts miss. Grafana-described proposal, not a measured result.

Language-model agents can communicate through continuous hidden states invisible in public transcripts. Verifiable Latent Alignments (VLA) links each private latent-state record and channel status to the resulting public action through a shared event identifier, and layers representation anomaly detection on top.

A complete trace of what agents said can still miss what they communicated. In multi-agent systems, message-and-tool-call spans are the wrong unit when coordination happens below the message layer.

"Evaluating Offline Monitoring of Internal AI Agents" studies monitoring agents from logged activity after the run, rather than with live instrumentation.

This matters when runtime tracing is incomplete or the agent runs where you can't watch it. The open question is how good such monitors are, which is itself an evaluation problem.