Agent engineering · know-how
🛠️ Solution·active·7 entries·7 sources·updated 2026-10-11
Generate several candidate tokens cheaply with a small draft model (or a lightweight head), then let the full model verify them in a single parallel forward pass. Accepted tokens come "for free," so latency drops without changing the output distribution. It attacks the strictly sequential, one-token-at-a-time decode that dominates an agent's wall-clock.
Speculative decoding has moved from a research trick to a serving default. vLLM ships it as part of a new model's day-0 support, and production teams report state-of-the-art latency by tuning the draft/verify pair to their own traffic.
The draft method is now a per-workload choice. Native MTP heads, trained EAGLE-3 drafts, and parallel drafters such as DFlash and DSpark all run on both NVIDIA and AMD accelerators. Benchmarks show the fastest method changes by model and task, with typical speedups of 1.6-2.9x. Vendor headline numbers (up to 15x) are ceilings, not planning figures.
Acceptance rate is the number that matters. The consensus practice is to tune num_speculative_tokens per workload and watch mean accepted length and per-position acceptance, not only end-to-end throughput. A poorly matched draft pays for drafting and verification and can come out slower.
The open problem is long agent context. Agents compress growing context to control cost and lose accuracy doing it. Asymmetric schemes, where a cheap drafter reads the full input and the verifier decodes a compressed view, are the current research answer. They are not yet a framework feature.
Lossless by construction: the full model still verifies every token, so quality is unchanged. The win depends entirely on acceptance rate. If draft and target disagree often (out-of-distribution inputs, a poorly matched draft model), you pay for the draft and the verify and can come out slower.
It costs extra memory and serving complexity: a second model or draft head to host and keep in sync with the target. The speedup is real on decode-bound, long-output work but marginal on short replies or prefill-bound prompts.
Treat it as a serving-layer knob tuned to the actual workload. Workload characterization (agent latency) and speculation are complementary, not alternatives.
It is one of the few latency levers that doesn't force a quality trade: the output matches what the target model would produce alone, so it is safe to enable broadly once the draft pairing is tuned.
For agent traffic, the same sequential decode is paid on every loop step, so the per-call saving compounds across a run. Validate it against your own traces before reaching for a smaller, lossy model, and add acceptance rate to your serving dashboards alongside TTFT and throughput.
Speculative decoding now ships in production stacks and in a new model's day-0 vLLM support, with gains tuned to each team's own traffic.
vLLM v0.26.0 adds the Inkling model family with a full support stack in the first release: base modeling, piecewise CUDA graphs, LoRA, NVFP4 quantization, and MTP=1 speculative decoding.
Speculation is now planned into a model's launch rather than added in a later optimization pass. When you adopt a new model on vLLM, check whether its speculation path is already supported before building your own draft setup.
Modal and Decagon describe how they reached state-of-the-art inference latency in production with speculative decoding, by tuning the draft/verify pair to Decagon's actual traffic rather than using a stock configuration.
The write-up frames speculation as a deployable latency win you can reproduce on your own stack, not a benchmark result. The lesson for builders is that the speedup depends on the workload, so measure it on your traces.
MTP, EAGLE-3, DFlash, and DSpark now run on both NVIDIA and AMD GPUs; the fastest method changes by model and task, so tune and measure acceptance per workload.
A measurement study of lossy speculative decoding methods finds a security-utility asymmetry: as inference gets faster, jailbreak and prompt-injection attack success rises much faster than utility degrades. The authors trace the loss to early tokens proposed by the draft model.
Their fix, SecureSD, applies a stricter verification criterion to draft tokens at early decoding positions, and reports better security at maintained efficiency and utility (paper-reported, no deployment numbers).
For builders: if you enable lossy acceptance to buy throughput, rerun your injection and jailbreak suite on the speculative configuration, not just the base model.
Also relevant to Untrusted input and tools can hijack an agent, Agent loops multiply per-call latency into slow, expensive runs
vLLM's guide to speculative decoding on AMD Instinct GPUs compares native MTP, Gemma 4's paired MTP checkpoint, EAGLE-3, parallel-draft DFlash, and DSpark (DFlash plus a Markov head). The winner changes by model and benchmark:
Tune num_speculative_tokens per workload (DFlash typically peaks near N=7). Watch mean accepted length and per-position acceptance rate, not only end-to-end throughput, to catch a pairing that costs more than it saves.
AMD's Quark toolkit now trains, quantizes, and serves EAGLE-3 draft models with vLLM on Instinct GPUs. AMD reports up to 2.00x throughput for Kimi-K2.5 and 1.79x for MiniMax-M2.5.
Draft-and-verify is no longer tied to one vendor's silicon. Teams running mixed or non-NVIDIA fleets can expect the same serving knob to exist on both, though the draft model still has to be trained for each target.
NVIDIA reports up to 15x inference-performance gains on Blackwell GPUs using DFlash, a speculative-decoding method that drafts several tokens in parallel rather than one at a time.
Treat the number as a vendor-reported ceiling, not a planning figure. The useful signal is that accelerator vendors now tune draft-and-verify for their own hardware instead of leaving it to the serving framework.
Research now lets drafter and verifier see different context, aiming to keep full-context accuracy while the expensive model decodes a compressed view.
AsymSpec drops the assumption that drafter and verifier share identical context. A lightweight drafter reads the agent's full, uncompressed input; the large verifier works from a compressed view. Contrastive δ-fusion of logits and a divergence-aware acceptance gate keep verification stable.
The paper reports roughly 90% of full-context accuracy at 1.3-1.7x throughput and 0.2-0.3x the compute of full-context decoding. That targets the agent trade-off where compressing a growing context to control cost usually costs task accuracy too. It is a research result, not yet a serving-framework feature.