Agent Builder Foundations

Agent-driven load compounds: engineers ship more changes, and each change generates more downstream work. A service built for human-paced load, especially one with a single writer, degrades from slow to silently wrong before anyone sees it is out of headroom. Capacity patches buy shrinking runway against that curve. The fix that held was structural: stateless, append-only ingestion, with aggregation moved to a separate layer that scales horizontally.

• One unshardable component is the bottleneck. In Anthropic's test-selection service, only the single-writer listener needed to change; the selector and the rest of CI were fine.
• Lag turns into data loss at high throughput. Twenty minutes behind meant tens of thousands of missing results, so real failures merged and unrelated tests looked flaky.
• Shrinking runway is the signal. Bigger machine, then sharding, then forced restarts: about 70 days, 29 days, under a day. Each patch moved the bottleneck without removing it.
• Decouple ingestion from aggregation. Stateless workers journal results; a separate consumer rolls the journal into history. It costs more to run than one writer, but scaling becomes a knob instead of a rebuild.

Test-impact analysis needs a history of which tests exercised which code and a fast way to select the minimal test set for a change. The failure was in writing that history, not in selection. A single writer applying every result is simple and correct at low volume, but its ceiling is one process's write rate, regardless of hardware elsewhere. As input grows, the gap between "result happened" and "result recorded" widens, and every reader of the record works from data that is increasingly wrong, not merely slow.

The redesign is a general pattern. Recording an event becomes append-only and shardable: any worker accepts and journals any result, holding no state. Aggregating events into queryable history becomes a separate consumer that rolls the journal forward every few seconds. Ingestion loses its single-process ceiling, and aggregation lag becomes a tunable staleness bound instead of an unbounded backlog.

• Find single-writer components whose input scales with agent output: results listeners, vector-index writers, eval aggregators, feedback-label stores.
• Design v0 for 10-20x the scale you expect, Anthropic's stated lesson, because agent-authored output compounds faster than headcount.
• Treat lag as a correctness bug. Alert against a hard staleness bound once a stale read can cause a wrong decision, such as skipping a test that should run.
• Stop patching at the second or third capacity fix for the same bottleneck. Shrinking runway means the architecture is the constraint.
• Default to append-only ingestion plus separate aggregation for services that must keep up with agent-driven event volume.

• Adding capacity to the same design and getting less runway each round.
• Treating aggregation lag as a performance metric while downstream consumers make wrong decisions on stale data.
• Planning capacity on headcount or feature velocity instead of agent-authored output.
• Assuming a service that worked before agents wrote much code will keep working as that share grows.

See agent cost and agent reliability for operating agent-driven pipelines at scale, and agent tracing for observability that surfaces growing lag before it causes undetected failures.