Agent Builder Foundations

A cheap draft process proposes several tokens ahead of where the target model has actually decoded. The target model then checks all of those guesses in a single forward pass instead of one at a time. Tokens the target model would have produced anyway are accepted for free; the first guess it disagrees with is thrown out and resampled correctly from there. The accept/reject math guarantees the final output has exactly the same distribution as running the target model alone — the speedup comes from turning a sequential dependency into a parallel check, not from approximating the model.

Decode at typical serving batch sizes is memory-bandwidth-bound, not compute-bound: most of a decode step is spent streaming model weights through the GPU, with FLOPs to spare. Speculative decoding spends that spare compute. Verifying several draft tokens in one forward pass costs about the same wall-clock as producing a single token normally, because the bottleneck — memory traffic — barely grows while the extra parallel compute is nearly free, right up until the batch gets large enough to saturate the chip.

One decode cycle has three steps:

• Draft. A cheap process — a much smaller separate model, or a lightweight head trained alongside the target model — autoregressively proposes several candidate tokens ahead of the current position.
• Verify. The target model runs a single forward pass over the drafted tokens together, producing next-token probabilities at every position at once. This is what the parallelism buys: a transformer's forward pass is inherently parallel across positions — only its own token-by-token *sampling* was ever sequential.
• Accept or reject. Walk the candidates left to right. Accept a draft token with probability min(1, p_target(x) / q_draft(x)). At the first token the check fails, discard every draft token after it and resample that position from the corrected residual distribution, then start the next draft cycle from there.

The engineering advances that made this a serving default (EAGLE and its EAGLE-3 successor, now served in vLLM on AMD Instinct at about 2x throughput) are almost entirely in the draft step: instead of an independently trained small model, they train a lightweight head to predict the target model's own next hidden-state features autoregressively. Because the draft is derived from the target's own internal state rather than a separately trained approximation, its guesses correlate far more with what the target model would actually choose, which is what raises the acceptance rate high enough to pay for the extra verify pass.

The correctness argument is a modified rejection sampling scheme. For a draft distribution q and target distribution p, accept a drafted token x with probability min(1, p(x)/q(x)). If rejected, resample from the residual distribution max(0, p(x) - q(x)), renormalized. Summing the accept branch and the reject-then-resample branch over all possible tokens reproduces p(x) exactly, regardless of what q is — the draft can be a bad guesser and the math still holds, it just accepts less often. The draft is *free information*, never *different information*: a bad draft costs you speed, not correctness.

The speed argument follows from the memory-bound builder model above. If one decode step costs time T largely independent of small compute increases, then verifying k draft tokens in one step still costs roughly T, not k · T. Accepting an average of α · k tokens per verify step (α = acceptance rate) turns α · k sequential steps into one, so the achievable speedup tracks 1 + α · k — until batch size grows enough that the GPU becomes compute-bound and the "T regardless of k" assumption stops holding.

• Measure acceptance rate on your own traffic before trusting any vendor's throughput multiplier. Low-entropy, structured output — code, JSON, repetitive tool-call arguments — accepts at much higher rates than open-ended creative text, so the same technique can be a large win for one agent and a marginal one for another.
• Prefer a feature-level draft head trained alongside the target model (the EAGLE approach) over an off-the-shelf small model as the draft; acceptance rate is the entire economics of the technique.
• Benchmark end-to-end latency and cost per token at your real concurrency, not tokens/sec at one batch size — the win shrinks as concurrency rises and the GPU shifts from memory-bound to compute-bound.
• Treat vendor multipliers (2x, 15x) as workload- and hardware-specific ceilings, not a number you inherit automatically; validate against your own agent's decode traffic.
• Because the technique is lossless by construction, it is safe to roll out broadly once tuned — there is no accuracy eval cycle to run, only a latency/cost one.

• Assuming the speedup is free at any batch size. At high concurrency, verify compute competes with the batch; the benefit shrinks or vanishes once decode is compute-bound.
• Mismatched draft and target. Swapping in an unrelated small model, or reusing a draft head trained against a different target checkpoint, tanks the acceptance rate and can make decode slower than not speculating at all, since you still pay the draft cost with little payoff.
• Treating headline multipliers as guaranteed. A hardware-co-designed benchmark number reflects a specific model, batch size, and accelerator; a different production workload will see a different number, often much smaller.
• Believing it changes output quality. It is lossless by construction; a quality change after enabling it means a bug in the accept/reject implementation.

See speculative decoding for current vendor and hardware coverage, and agent latency for why the strictly sequential decode loop this technique attacks dominates an agent's wall-clock in the first place.