Agent Know-How

A from-scratch Swift port of vLLM's continuous batching onto an iPhone's MLX kernels left-pads late-arriving requests and merges them into a shared KV-cache offset. It hit 169 aggregate tokens/sec across 8 concurrent streams versus llama.cpp's 90.

A 16-request, ~17K-prompt-token multi-agent workload ran in 25 seconds without thermal throttling; llama.cpp needed 47 seconds for half the load. Continuous batching answers concurrent decode on-device too, not only in the data center.

At 5,000 GitHub stars, vLLM Semantic Router sets its next phase: building the training, evaluation, and inference engine for a Mixture-of-Models era.

"Which model handles this request" becomes a first-class serving-layer decision with its own eval loop, not a routing feature bolted onto an engine. For agents, that is where fast-and-cheap versus slow-and-capable gets decided per call.

SMetric finds agent traffic already has over 80% KV-cache reuse in production, but generic schedulers over-index on cache locality and let load imbalance cap cluster throughput. Agents act only on complete responses, so cluster tokens per second matters more than per-token latency.

It load-balances the first hop of each agent session, then routes every later request cache-aware. Reported gains versus prior schedulers: 10-16% throughput under prefill-decode colocation and 2-34% on prefill under disaggregated serving.

Static batching policies need manual tuning per traffic shape and cannot adapt when patterns shift. This paper trains REINFORCE and PPO agents to learn batching and routing policies, modeling queue state, request type, and GPU availability on a discrete-event simulator validated against queuing theory and production traces (Azure Functions, BurstGPT).

Agent tool-calling produces exactly this bursty, heterogeneous load, not the steady arrival rate a chat workload has.

vLLM Semantic Router turns its vllm-sr/auto routing into a bounded micro-agent runtime: confidence scoring, ratings, model fusion, and workflows run inside the serving layer instead of in a separate orchestration hop above it.

Each orchestration hop removed is a full model call's latency the agent no longer pays. The serving layer is starting to absorb logic that used to live in the application.

TraceLab (University of Washington) characterizes real, day-to-day coding-agent usage across multiple agents and model families for serving-system research. Existing public traces and benchmarks don't capture that usage.

The premise: agent workloads don't look like chat. Coding agents issue bursty, long-context, tool-interleaved requests, so a server tuned on a generic chat trace is tuned for the wrong shape.

Prefix caching assumes reused text sits at the start of the prompt. RAG assembles different chunks per query, and a multi-agent coordinator reads other agents' reports, so reusable text lands mid-prompt, at the wrong positions, sometimes from another checkpoint.

KVShareArena scores reuse methods on that case. Correcting positions alone suffices until a question needs several retrieved sources; then only methods that pay for partial re-encoding or extra training recover half to two-thirds of the gap. An unrepaired cache can score worse than no cache, and compression that looks harmless on one prompt falls behind plain position correction.

vLLM integrates HiSparse as a pressure-driven memory tier that composes with the Hybrid Memory Allocator and KV offloading. It activates once a request's KV state no longer fits in GPU memory, so GLM 5.3 requests keep decoding instead of stalling or falling back to a slower path.

Concurrency stays high exactly where the storage-bandwidth bottleneck would otherwise cap it.

A universal context-reuse layer translates the KV state one model produced into a form a different model can consume, skipping the second model's prefill.

• Same family (Qwen2.5-7B to 1.5B): LongBench2 accuracy rises from 27.59% to 34.48%.
• Across families (Qwen2.5-1.5B to Gemma-2-2B): up to 67.05% less prefill cost at 4K context.
• Large to small (Llama3.1-70B to Qwen2.5-7B): end-to-end latency drops from 899ms to 138ms, at 44.0% vs 45.7% accuracy.

It fits multi-model pipelines where a cheap model would otherwise redo a bigger model's prefill.

vLLM-Omni's Distributed Layerwise Offload shards and streams diffusion-transformer weights across devices. It serves a measured 124 GB Cosmos3 model on 64 GB of HBM and estimates a path toward 200B+ parameter models.

It applies the offload-instead-of-fit approach to model weights rather than KV state: memory capacity, not compute, decides which models fit a given fleet.

vLLM's Decode Context Parallelism (DCP) shards the KV cache across GPUs along the sequence dimension. It reports 3x higher throughput on long-context agentic workloads versus standard tensor parallelism.

It attacks the storage-bandwidth bottleneck through parallelism rather than compression or offload: each GPU holds and reads only its slice of a long agent context.

OpenLake offloads KV state from GPU memory into a shared RAM and NVMe tier, with a CUDA kernel that losslessly compresses blocks before they leave the GPU. Its motivation: one 256K-token conversation on Gemma 4 31B produces about 43GB of KV state, more than half an 80GB H100.

Because the tier is shared, a prefix cached on one host is cheap to fetch from another instead of being recomputed. On a 128K-context workload, time-to-first-token fell from 44 seconds to 0.6 when the prefix was reused across hosts. The project claims 50% lower long-horizon inference cost.

RaBitQCache uses randomized rotated binary quantization to compress the KV cache, and an adaptive top-p token budget instead of a fixed top-k for sparse-attention retrieval, holding generation quality on long-context inference.

It cuts the memory I/O that DualPath names as the agentic bottleneck: a smaller cache is less state to move at every step.

DualPath finds that the binding constraint in agentic LLM inference is storage bandwidth, not compute: the agent's growing KV and context state has to be streamed back in at every step of the loop.

That changes the optimization target. Faster kernels help less than shrinking, offloading, sharding, or reusing the state each step has to move.

The vLLM AFD plugin runs attention and FFN computation on separate execution paths for MoE serving, with GPU and Ascend NPU backends, connector-based execution, and graph and micro-batching ("ubatching") support.

Prefill/decode disaggregation splits a request by phase; AFD splits a single forward pass by compute type. That is a second knob for allocating hardware across the two halves of large open MoE models.

TileRT plugs a decode-only runtime into vLLM's prefill/decode split through the V1 connector interface, with zero changes to vLLM. KV state moves from stock vLLM prefill nodes to TileRT decode nodes over RDMA, and multi-token speculative decoding starts as soon as it lands.

It reaches peak decode throughput at a best-case 4.0-token acceptance rate on an 8-GPU setup. Today it is limited to one in-flight request per decode node and a narrow model list.

vLLM v0.25.0 makes Model Runner V2 the default execution path for all dense models and deletes the legacy PagedAttention implementation. MRv2 gains prefix caching for Mamba hybrid models and dynamic speculative decoding compatible with full CUDA graphs. Tool-call and reasoning-token parsing across model families is unified under one Streaming Parser Engine.

Teams pinned to older vLLM releases should plan the upgrade: the attention path the engine was first known for is gone.

vLLM's transformers modeling backend uses torch.fx graph analysis plus AST rewriting to fuse operations into optimized vLLM kernels automatically. It matches native per-model integration throughput on dense and MoE Qwen3 models without hand-written per-model code.

Keeping up with new architectures is a maintenance tax on latency. This lowers the cost of getting a new model onto a fast path.

Tencent's HPC-Ops integrates Hopper-optimized attention and FP8 MoE backends into vLLM for the Hunyuan Hy3 model on NVIDIA H20, improving mixed-length decode, MoE latency, time-to-first-token, and time-per-output-token.

Serving gains are increasingly model- and hardware-specific. The mixed-length, bursty decode it targets is the pattern agent loops produce, not the uniform batches a generic benchmark assumes.

NVIDIA's work with AWS across Amazon OpenSearch and Amazon EC2 is framed around four constraints: low-latency inference, fast vector search, GPU price-performance, and infrastructure that grows without multiplying operational complexity.

Infra vendors now sell explicitly on latency. For an agent, raw engine speed moves only one term of the budget; loop length and tool round-trips still multiply it.

Modular 26.4 ships mixture-of-experts serving that the company calls state of the art, alongside model bring-up via agent skills and Mojo 1.0 Beta 2. The performance claim is vendor-reported.

It is one more engine competing on MoE decode, the architecture most large open-weight releases now use. Measure it on your own agent traffic before switching engines.

vLLM v0.28.0 continues the Kimi K3 optimization stack-wide: Decode Context Parallel support, fused FlashKDA decode and prefill kernels, combined all-gathers with a 1.5-3x kernel-level speedup, an adaptive speculative token budget for about 60% better DSpark time-to-first-token, and shared-expert sharding that saves about 17 GiB per GPU.

It also lands DeepSeek V4's sparse MLA end-to-end across plain decode, MTP, and DSpark speculative decoding. The fast path now covers every decode mode the model runs.

vLLM's day-0 Kimi K3 serving keeps the preview's hybrid KDA prefix caching and adds DSpark speculative decoding, production-scale disaggregation, and optimized kernels across NVIDIA and AMD GPUs.

The gap between preview and production support closed within one release cycle. Latency-tuned serving on day one now holds across a model's preview-to-GA transition, not just its announcement.

vLLM v0.26.0 ships the Inkling model family with piecewise CUDA graphs, Hopper FA4 relative attention, MTP=1 speculative decoding, LoRA, and NVFP4 quantization in one release.

A DeepSeek-V4 performance push adds a specialized routing kernel (2.94% end-to-end TPOT), fused top-k bias (1.5-2x at kernel level), and redundant-copy removal (1.8% TPOT) without touching the serving architecture. Small per-step gains like these compound across every step of an agent loop.

vLLM now runs end-to-end on pre-release NVIDIA Vera Rubin hardware.

Day-0 support now covers GPU generations, not only models. Teams planning capacity on the next hardware generation can expect the open engine to be ready at launch.

vLLM's preview of production-scale Kimi K3 support includes KDA-aware prefix caching, fused kernels, optimized MXFP4 MoE, multimodal integration, and initial NVIDIA and AMD paths.

It lands ahead of a full optimization pass, extending the day-0 pattern to a new open-weight architecture with its own attention variant.

vLLM shipped full-feature support for Thinking Machines' 1T-parameter multimodal Inkling the day it released, with MTP, long-context serving, and parallelism. It reaches 380 tokens/sec per user with speculative decoding versus 140 without on 4 NVIDIA GB200 GPUs.

New models now get latency-tuned serving at launch, so speculative decoding and parallelism apply from the first day instead of after a follow-up optimization pass.

vLLM v0.24.0 (571 commits from 256 contributors) adds MiniMax-M3 with a fast follow-on BF16/FP8 indexer, MXFP4 support, FP8 sparse GQA, and AMD/ROCm tuning: MXFP8 MoE on gfx950, FP8 per-channel weights on MI300X, and an FP8 KV-cache fix. It also fixes a MiniMax-M2 performance regression and keeps maturing DeepSeek-V4 support.

New open-weight models arrive with quantized, vendor-specific fast paths in the same release, so self-hosters don't wait for a later optimization pass.

AsymSpec drops speculative decoding's requirement that drafter and target share identical context. A lightweight drafter reads the agent's full, uncompressed input while the large verifier decodes from a compressed view, with a divergence-aware acceptance gate keeping verification stable.

It recovers about 90% of full-context accuracy at 1.3-1.7x throughput and 0.2-0.3x the compute of decoding on full context. That eases the usual trade between compressing a growing agent context for latency and the accuracy compression costs.

DARTree extends a pretrained autoregressive correction head for diffusion drafters from single draft chains to draft trees, scoring and pruning candidates across the whole tree in one batch.

Across seven math, code, and chat benchmarks it accepts up to 12.97 tokens per verification round (98.6% more than DFlash, 27.9% more than Domino), for up to 9.73x lossless speedup over plain autoregressive decoding. It is training-free. Mechanics: speculative decoding.

Kog built Laneformer 2B as a latency-first model, designed together with the Kog inference engine rather than adapted to a fast server after the fact.

Most of an agent's calls don't need frontier breadth. Sending the bulk of them to a small, predictable model is the same downshift that controls cost, applied to wall-clock time.

Cube argues that a single agent request fans out into tens of database or API queries, and a multi-step workflow into hundreds. Each inherits chat-era expectations: a few hundred milliseconds feels responsive, a couple of seconds feels broken.

The dashboard semantic-layer pattern (pre-aggregated rollups served through query rewriting, columnar storage with partition pruning) is being repurposed as agent infrastructure because it was built for many small interactive queries, not a few large batch ones.

Modern Treasury runs checks on the developer's machine instead of round-tripping to a remote CI runner, cutting feedback latency for both human developers and coding agents.

For a coding agent, waiting on a CI runner sits on the same wall-clock budget as each model call. A slow tool round-trip can dominate the loop even when inference is fast.

Loka built a voice agent on Amazon Nova 2 Sonic to fix a specific failure: slow, robotic voice assistants make customers hang up, which damages the brand and drives up support costs. The architecture treats round-trip time as a first-class constraint from the start, not a tuning pass afterwards.

Interactive modes set a hard latency floor. A voice agent's budget is set by when the caller gives up, not by what the serving stack happens to deliver.