LLM Digest
Subscribe

Agent Know-How

Agent engineering · know-how

에이전트 실행 추적 및 분석

🛠️ Solution·active·12 entries·12 sources·updated 2026-11-04

Capture every agent run as a structured trace — the prompts, tool calls, results, retries, and sub-agent handoffs — in a common format, then analyze those traces to find what broke and why. Tracing is the substrate that makes an agent debuggable, evaluable, and operable instead of a black box that occasionally misbehaves.

Latest updates
State of the art

Tracing now has three layers, and each moved this year.

Capture is commoditizing. OpenTelemetry/OpenInference-style span schemas are the common format, zero-config SDKs instrument LLM calls without code changes, and serving platforms such as Cloudflare emit agent spans (invoke_agent, execute_tool, tool_approval) natively. Coverage is widening past text to voice agents. The catch is in the defaults: payload recording, truncation, and billing differ per integration, not per vendor.

Storage is becoming real infrastructure. Traces are large, nested JSON documents, and searching them at fleet scale takes purpose-built indexes (an inverted index over object storage) or one store for traces and eval results. The self-owned alternative is a lossless archive in your own bucket, searchable and exposed to agents over MCP.

Analysis is where the movement is. The pattern is trace-in, explanation-out: a model reads the trace corpus, clusters recurring failures, and proposes harness fixes, or triages an alert from live traces. Viewers are getting more readable too: chat-style session threads instead of span trees, and inline visualizations that render the actual query result so an engineer can check what the agent claims.

The open problem is completeness. Traces are not lossless when payloads get truncated, model-over-trace analyzers can miss the rare failure, and agents in a multi-agent system can coordinate through hidden states that no message-and-tool-call span records (see multi-agent and agent observability).

Trade-offs

Tracing adds instrumentation overhead and storage, and high-cardinality traces get expensive to retain and search at fleet scale, so retention, sampling, and PII scrubbing are real decisions. They are not even consistent within one platform: payload defaults can differ by SDK, so "does this capture PII by default" is answered per integration, not per vendor.

Model-over-trace analysis is its own LLM cost and reliability line item; the analyzer can be wrong or miss the rare failure. A vendor trace format can lock you in, while plain JSONL is portable but shifts the analysis burden onto you.

Best value comes from standardizing the capture format early, so the storage and analysis layers, homegrown or managed, stay swappable.

Why it matters for platform engineers

Traces are the agent equivalent of logs and metrics: the precondition for evaluation (you grade trajectories you captured), for cost control (per-step token attribution), and for incident response (a replayable run). Owning a portable trace format and an analysis loop is the difference between operating an agent and guessing at it.

Developments 12

Capturing the run: zero-config SDKs, platform spans, new modalities

4 entries · latest Nov 4, 2026

Capture is commoditizing into drop-in SDKs and platform-native spans and now covers voice; payload defaults, truncation, and per-span billing differ by integration, and latent multi-agent channels escape the span schema entirely.

Claude Platform exports native OpenTelemetry traces to Databricks MLflow

Databricks announced that MLflow is now a supported destination for native OpenTelemetry export from the Claude Platform. Traces of model and tool calls land beside existing MLflow data, so teams can find recurring failures, such as an agent rereading the same source, and test whether a change helps.

For builders: OpenTelemetry export means trace capture needs no custom SDK, and the backend stays swappable. This is a vendor announcement; check span coverage on your own agents.

Cloudflare adds agent spans to Workers traces, with truncation and uneven payload defaults

Cloudflare's agent tracing adds invoke_agent → chat/execute_tool → tool_approval spans to existing Workers traces, keyed by agent name, agent ID, and conversation ID. Sessions replay turn by turn.

The caveats are the news:

  • Payload recording defaults differ between its two supported SDKs (off in one, on in the other).
  • The docs warn traces are not lossless; payloads may be truncated.
  • From October 1, 2026, every span counts as a billable event.

Check defaults per integration: one stack can over-retain personal data while another drops the tool arguments a debugging session needs.

Show 1 earlier entry

Storing and searching traces at fleet scale

3 entries · latest Aug 20, 2026

Vendors are hardening trace stores with inverted indexes over object storage and one table for traces plus evals, while self-owned archives in your own S3 bucket keep sessions portable.

Pond archives agent sessions losslessly into a team's own S3 bucket

Pond stores agent sessions from several machines in one S3 bucket with no database service to run, adds search over the archive, and exposes the archive to the agent as an MCP server.

It is the self-owned counterweight to vendor trace stores: the portable-file position plus the multi-machine collection and search it was missing. It treats sessions as an asset to keep, not a debugging byproduct that ages out with a retention window. It is an early Show HN project.

Langfuse v4 stores traces and eval results in one immutable ClickHouse table

Langfuse v4 rebuilds both agent traces and evaluation results onto one immutable ClickHouse table, collapsing what were separate storage paths into one queryable store.

A score can now be joined back to the run that produced it without crossing systems. It is the same infrastructure hardening as LangSmith's SmithDB, aimed at unifying capture and evaluation rather than indexing traces alone.

SmithDB full-text-searches nested agent traces in object storage at 400 ms P50

LangChain's SmithDB builds a custom inverted index over object storage so agent traces can be full-text searched and JSON-filtered directly. It reports a 400 ms median (P50) query latency, even though each trace is a large, deeply nested JSON document.

At fleet scale, "traces are stored somewhere" is not enough: search over payloads is what makes them usable in an incident. This is the bar a self-hosted trace store has to meet.

Reading and diagnosing traces

5 entries · latest Sep 24, 2026

Tools now run a model over trace corpora to cluster failures and triage alerts, and viewers are moving to chat-style session threads and inline, query-backed visualizations.

OpenSearch MCP Apps render trace and log visualizations inline in the agent chat

Amazon OpenSearch Service's MCP Apps return interactive visualizations alongside an agent's text response, from one locally run MCP server. An alert investigation (trace lookup, log-pattern clustering, distributed-trace analysis, RED metrics, service-dependency mapping) happens in one IDE chat thread.

The visualization renders the actual OpenSearch query result, not a model-generated chart. An engineer can verify each step the agent claims without re-running the query in a separate dashboard.

Show 2 earlier entries

LangSmith adds an on-call copilot that triages fleet alerts from live traces

LangSmith's June 2026 release adds a Fleet on-call copilot for alert triage, voice trace debugging, and experiment status tracking.

Trace reading becomes an assistive workflow: the copilot starts from the alert and the traces behind it instead of an engineer scrolling spans. Like any model-over-trace analyzer it can be wrong or miss a rare failure, so treat its triage as a lead, not a verdict.

HALO runs a model over trace corpora to find recurring harness failures

HALO is an open-source, local tool that ingests Langfuse, Arize/OpenInference, or JSONL traces and uses an RLM-based engine to find recurring harness-level failure modes across runs. It writes reports you can hand to Cursor or Claude Code to patch the agent, and its desktop app answers questions over your traces.

The trace becomes the input to a diagnosis loop, not just an audit log. Because it reads open trace formats, the analyzer stays swappable.

From related topics
  1. Agents plan multi-step work badly — they loop, stall, or skip stepsHarness changes alone lifted LangChain's coding agent from 52.8% to 66.5% on Terminal Bench 2.0
  2. Coordinating multiple agents adds more failure than capabilitySpotify Ads AI enforces one agent, one package, one owner, with deterministic gates outside the agents
  3. Coordinating multiple agents adds more failure than capabilityAgents can coordinate through hidden states that no transcript or span records
Evidence · 12 sources