Agent Know-How

Speculative decoding has moved from a research trick to a serving default, and the recent work is about making the draft step both cheap and accurate enough that the acceptance rate justifies the extra verify compute. Modal and Decagon report state-of-the-art inference latencies in production by tuning the draft/verify pair to their workload, framing it as a practical, deployable win rather than a benchmark curiosity. On the hardware side, NVIDIA's DFlash pushes the technique into the silicon — up to ~15× inference-performance gains on Blackwell — showing the draft-and-verify pattern is being co-designed with the accelerator, not just layered on top in software. The throughline is that the gains are largest exactly where agents hurt most: long, latency-sensitive decode loops where shaving sequential steps compounds across every turn of the agent.

The hardware co-design push is no longer NVIDIA-only: AMD's Quark now trains, quantizes, and serves EAGLE-3 draft models with vLLM on Instinct GPUs, reporting up to 2.00× throughput for Kimi-K2.5 and 1.79× for MiniMax-M2.5 — evidence the draft-and-verify pattern is becoming a cross-accelerator serving default rather than a technique tied to one vendor's silicon.

Speculative decoding is also becoming a day-0 launch feature, not a follow-up optimization pass: vLLM v0.26.0 ships MTP=1 speculative decoding as part of the full support stack for its new Inkling model family from the first release, alongside base modeling, CUDA graph support, and quantization — the same "new model, latency-tuned serving on day one" pattern this page's throughline already tracks, now including the speculation setup itself instead of adding it later.

vLLM v0.26.0 ships MTP=1 speculative decoding for its new Inkling model family as part of the model's initial full support stack (alongside base modeling, CUDA graph support, and quantization) rather than as a later optimization pass — evidence that draft-and-verify setup is now planned into a new model's launch, not bolted on after.

Lossless by construction — the full model still verifies every token, so quality is unchanged — but the win is entirely a function of acceptance rate: if the draft and target disagree often (out-of-distribution inputs, a poorly matched draft model), you pay for the draft *and* the verify and can come out slower. It costs extra memory and serving complexity (a second model or draft head to host and keep in sync), and the speedup is real on decode-bound, long-output work but marginal on short replies or prefill-bound prompts. Best treated as a serving-layer knob tuned to the actual workload — which is why workload characterization (agent-latency) and speculation are complementary, not alternatives.

It is one of the few latency levers that doesn't force a quality trade — the output is identical to greedy/sampled decoding from the target model, so it's safe to enable broadly once the draft pairing is tuned. For agent traffic, where the same sequential decode is paid on every loop step, the per-call saving compounds across the run, making it a high-leverage default to validate against your own traces before reaching for a smaller, lossy model.