{"slug":"agentic-code-ci-scaling","title":"Why does AI-generated code overwhelm your CI system, and what actually fixes it?","question":"Why does AI-generated code overwhelm your CI system, and what actually fixes it?","summary":"Anthropic's CI job volume grew 25x in six months as Claude wrote 80% of code changes. The bottleneck was a single-writer test-selection service; three capacity patches bought 70, 29, then under one day until a stateless, journal-based redesign held.","status":"active","cluster":"operations","cluster_label":"Cost, latency, and operations","updated":"2026-10-06","latest_evidence_added":"2026-09-16","audience":"strong-software-engineer","math_depth":"","sections":[{"heading":"Builder consequence","html":"<p>If agents write a growing share of your code or pipeline output, your infrastructure will not scale linearly with that growth. Anthropic&#x27;s <strong>CI job volume grew 25x in six months</strong> once Claude wrote 80% of code changes. What nearly broke the pipeline was not compute but one single-writer service, and each capacity patch bought less time than the last.</p>"},{"heading":"Short answer","html":"<p>Agent-driven load compounds: engineers ship more changes, and each change generates more downstream work. A service built for human-paced load, especially one with a single writer, degrades from slow to silently wrong before anyone sees it is out of headroom. Capacity patches buy shrinking runway against that curve. The fix that held was structural: stateless, append-only ingestion, with aggregation moved to a separate layer that scales horizontally.</p>"},{"heading":"Builder model","html":"<ul><li><strong>One unshardable component is the bottleneck.</strong> In Anthropic&#x27;s test-selection service, only the single-writer listener needed to change; the selector and the rest of CI were fine.</li><li><strong>Lag turns into data loss at high throughput.</strong> Twenty minutes behind meant tens of thousands of missing results, so real failures merged and unrelated tests looked flaky.</li><li><strong>Shrinking runway is the signal.</strong> Bigger machine, then sharding, then forced restarts: about 70 days, 29 days, under a day. Each patch moved the bottleneck without removing it.</li><li><strong>Decouple ingestion from aggregation.</strong> Stateless workers journal results; a separate consumer rolls the journal into history. It costs more to run than one writer, but scaling becomes a knob instead of a rebuild.</li></ul>"},{"heading":"Mechanism","html":"<p>Test-impact analysis needs a history of which tests exercised which code and a fast way to select the minimal test set for a change. The failure was in writing that history, not in selection. A single writer applying every result is simple and correct at low volume, but its ceiling is one process&#x27;s write rate, regardless of hardware elsewhere. As input grows, the gap between &quot;result happened&quot; and &quot;result recorded&quot; widens, and every reader of the record works from data that is increasingly wrong, not merely slow.</p>\n<p>The redesign is a general pattern. Recording an event becomes append-only and shardable: any worker accepts and journals any result, holding no state. Aggregating events into queryable history becomes a separate consumer that rolls the journal forward every few seconds. Ingestion loses its single-process ceiling, and aggregation lag becomes a tunable staleness bound instead of an unbounded backlog.</p>"},{"heading":"How to apply","html":"<ul><li><strong>Find single-writer components whose input scales with agent output</strong>: results listeners, vector-index writers, eval aggregators, feedback-label stores.</li><li><strong>Design v0 for 10-20x the scale you expect</strong>, Anthropic&#x27;s stated lesson, because agent-authored output compounds faster than headcount.</li><li><strong>Treat lag as a correctness bug.</strong> Alert against a hard staleness bound once a stale read can cause a wrong decision, such as skipping a test that should run.</li><li><strong>Stop patching at the second or third capacity fix for the same bottleneck.</strong> Shrinking runway means the architecture is the constraint.</li><li><strong>Default to append-only ingestion plus separate aggregation</strong> for services that must keep up with agent-driven event volume.</li></ul>"},{"heading":"Failure modes","html":"<ul><li>Adding capacity to the same design and getting less runway each round.</li><li>Treating aggregation lag as a performance metric while downstream consumers make wrong decisions on stale data.</li><li>Planning capacity on headcount or feature velocity instead of agent-authored output.</li><li>Assuming a service that worked before agents wrote much code will keep working as that share grows.</li></ul>"},{"heading":"Related","html":"<p>See <a href=\"/topic/agent-cost\">agent cost</a> and <a href=\"/topic/agent-reliability\">agent reliability</a> for operating agent-driven pipelines at scale, and <a href=\"/topic/agent-tracing\">agent tracing</a> for observability that surfaces growing lag before it causes undetected failures.</p>"}],"evidence":[{"id":"anthropic-2026-ci-test-impact-analysis","kind":"primary-doc","tier":"primary-doc-backed","title":"Agentic coding is straining CI. Here's how we scaled test impact analysis at Anthropic","note":"Anthropic's account of its own CI: job volume up 25x in six months, Claude writing 80% of changes, 8x more code per quarter. A single-writer test-selection listener fell behind; 20 minutes of lag left tens of thousands of results unapplied, letting failures merge. Patches lasted about 70 days, 29 days, then under a day. A stateless, journal-based redesign, built by one engineer in three weeks, flattened the backlog. First-party, single-company report.","added":"2026-09-16","url":"https://claude.com/blog/agentic-coding-is-straining-ci-heres-how-we-scaled-test-impact-analysis-at-anthropic","sid":"86abcc6f17fc520e"},{"id":"agentic-code-ci-scaling-editorial-synthesis","kind":"editorial-inference","tier":"editorial inference","title":"LLM Digest synthesis","note":"The numbers are Anthropic's own pipeline. The lesson extends to any stateful aggregation service in an agent-driven critical path, such as a vector-index writer, feedback-label store, or eval-result aggregator: a single-writer or unshardable component whose input rate scales with agent output, not headcount.","added":"2026-09-16"}],"related_topics":[{"slug":"agent-cost","title":"Agent token costs are unpredictable and easily run away"},{"slug":"agent-reliability","title":"Agents give fluent, confident-looking output even when it's wrong"},{"slug":"agent-tracing","title":"Tracing and trace analysis for agent runs"}],"related_playbook_cards":[],"related_storylines":[]}