Speculative decoding is often assumed to be fast because it does less work per token. That is not true. The total number of floating point operations (FLOPs) actually goes up for the same sequence. The speedup comes from changing the kind of work the GPU performs.
Decoding one token at a time is usually memory bound. For every token, the GPU must stream all active model parameters, such as the 27B parameters of Qwen3.5-27B, from global memory. Meanwhile much of the compute sits idle. Large batched decoding of thousands of tokens is the opposite: it is compute bound, with the GPU busy on tensor operations.
Speculative decoding uses a small draft model to propose a sequence of likely tokens, which the big model then verifies in a single forward pass. At low batch sizes, this turns otherwise wasted compute into progress. Instead of running five sequences each advancing one token, one sequence can advance five tokens at once, using the same total work.
The technique has limits. If the acceptance rate is below 100%, every rejected token is wasted work. At high throughput, when the GPU is already compute bound, speculation adds cost without benefit, which is why vLLM offers the --speculative-disable-by-batch-size setting to turn it off.
Source: Hacker News · Summarized by HeadlinesBriefing