LLM Digest
Subscribe

Agent Know-How

Agent engineering · know-how

Agent infrastructure buckles under concurrency, not just load

🧱 Obstacle·scalability·active·4 entries·4 sources·updated 2026-10-06

A single agent is a serving problem; a fleet of agents is a distributed-systems problem. Every agent that spins up a sandbox, holds a serving slot, or opens a session adds concurrent, short-lived state. The coordination layer that tracks that state, not raw compute, is what breaks first at agent-native scale.

Latest updates
  1. GKE Agent Substrate targets 10x sandbox density with sub-500ms resume, and GKE scales to zero
  2. Stateless MCP removes sticky sessions, so remote MCP servers scale like any stateless service
  3. vllm-metal brings paged KV cache and packed batching to Apple Silicon for concurrent agent load
  4. Modal drops central coordination to reach 1 million concurrent sandboxes in under 60 seconds
State of the art

The consensus fix is to remove the central coordinator, not scale it. The same move now shows up at three layers:

  • Sandbox scheduling. A Kubernetes-style scheduler backed by strongly consistent state caps out before the hardware does. The working design lets each worker accept or reject work on its own resources and publish state to a shared stream. Modal reports 1 million concurrent sandboxes in under 60 seconds this way.
  • Inference serving. Paged KV caches and packed, ragged batching stop padding overhead from compounding as concurrent requests share a batch.
  • Tool protocol. Stateless MCP drops sessions and sticky routing so remote servers scale like any stateless service.

Density is the second lever. Instead of keeping sandboxes warm, platforms suspend idle ones and resume them in under a second, and Kubernetes can now scale agent workloads to zero.

The open problem is where the state goes. Removing the coordinator or the session does not remove the state. Retries, idempotency, application state, and observability move to layers the platform team now owns. Most published numbers are vendor-reported and not independently reproduced.

Why it matters for platform engineers

Concurrency fails differently from throughput, and it fails first in the layer teams rarely load-test: the scheduler or coordinator that tracks which sandbox, session, or KV-cache page belongs to which agent. A platform that serves one agent well can still fall over when a thousand run at once.

Budget for concurrent state (sandbox lifecycle, session tracking, KV-cache admission), not just aggregate request volume. Prefer designs where each worker decides locally over ones that route every request through one coordinator. See agent sandboxing for the isolation side of the same fleet problem.

Developments 4

Removing the central coordinator instead of scaling it

2 entries · latest Sep 27, 2026

Sandbox schedulers and the MCP protocol both drop central, strongly consistent coordination so each worker or request decides locally and scales horizontally.

·1 source

Stateless MCP removes sticky sessions, so remote MCP servers scale like any stateless service

AWS details how the latest MCP specification removes protocol-level sessions, sticky-session requirements, and session storage for remote MCP servers. Requests can route independently, so servers scale horizontally behind an ordinary load balancer.

It is the same "remove the coordinator" move Modal made for sandbox scheduling, applied to the tool-calling protocol. The state does not disappear: application state, retries, observability, and idempotency move to other layers that you now own. See MCP for the full protocol change.

·1 source

Modal drops central coordination to reach 1 million concurrent sandboxes in under 60 seconds

Modal rebuilt its sandbox infrastructure after Kubernetes-style orchestration capped out: etcd can't be sharded within a keyspace, and scheduling work grows with container and node count, so the coordinator became the bottleneck before the hardware did.

The rebuild removes it. A horizontally scaled fleet of scheduling servers load-balances requests, each worker accepts or rejects a sandbox creation over RPC based on its own free resources, and workers publish state to one Redis stream the team expects to hold past 100,000 workers. Reported results: 1 million concurrent sandboxes in under 60 seconds, about 50,000 creations per second, and median cold start under 0.5 seconds.

Serving many agents at once without padding waste

1 entry · latest Sep 23, 2026

Paged KV caches and packed, ragged batching keep time to first token flat as concurrent agent requests share a batch, now down to Apple Silicon.

·1 source

vllm-metal brings paged KV cache and packed batching to Apple Silicon for concurrent agent load

vllm-metal brings vLLM's scheduler, paged KV cache, and continuous batching to Apple Silicon (M1 Pro through M5 Pro). It replaces padded batching with packed queries, concatenating request tokens into one ragged batch instead of padding each request to the same length. The release also adds batched MTP and automatic M5 prefill acceleration, and reports flatter TTFT under concurrent agent load.

Padding overhead grows with every request sharing a batch, so this is a concurrency fix, not a raw-speed one. Paged block tables let variable-length sequences share memory without reshaping the cache, which is what makes admission control workable under concurrent load.

Sandbox density and scale-to-zero for idle fleets

1 entry · latest Oct 3, 2026

Platforms now suspend idle agent sandboxes and resume them in under a second, so idle fleets stop holding capacity; density claims are vendor-reported.

·1 source

GKE Agent Substrate targets 10x sandbox density with sub-500ms resume, and GKE scales to zero

Google's September AI-infrastructure update adds the open-source GKE Agent Substrate, aimed at millions of sandboxes at 10x the density of standard container runtimes. Google claims sub-500ms resume, over 500 suspend/resume activations per second, and kernel- and network-level isolation. GKE also now scales workloads to zero natively, through the HPA with KEP-2021 support.

These are vendor-reported targets. The pattern they point to: suspend idle agent sandboxes and resume them on demand instead of keeping them warm, so an idle fleet stops holding capacity.

From related topics
  1. Agents forget across steps and sessionsPostgres as the agent memory layer: JSONB, HNSW vectors, and row-level locking
Evidence · 4 sources
All sources behind this page