TL; DR: speculative decoding से vLLM एकल target-model pass में multiple drafted tokens verify कर सकता है। हमारे प्रयोगों में, इसका effect output-token throughput drafting methods और proposal lengths के अनुसार बदलता है, और bhi model family, draft checkpoint, workload, aur acceptance behavior par bhi depend karta hai।

परिचय: Large language models ek wide range of applications support karte hain, lekin unhe scale par serve karna careful optimization ki zaroorat hoti hai. Standard autoregressive decoding most LLM serving systems use ki base line hai: model ek token generate karta hai, usse sequence mein append karta hai, phir updated sequence use karke next token generate karta hai. ye process simple aur reliable hai, lekin serving loop abhi bhi ek ek committed token ko advance karta kyunki output tokens ko strict left-to-right order mein produce karna padta hai.

Speculative decoding [1] iske baseline par draft-and-verify mechanism build karta hai. A lightweight draft component candidate future tokens propose karta hai, aur target model unhe verify karta hai jab tak unhe commit nahi kareya jata. Kitne bhi draft tokens accept ho jayenge, system ek single target-model verification step se multiple output tokens commit kar sakta hai target model output behavior preserve rakhte hue.

Yeh post explore karta hai ki vLLM mein speculative decoding kaise kaam karta hai aur hamare test environment se nikalne wale measurements. Ham first autoregressive decoding baseline aur draft-and-verify process ka review karenge. Phir ham pांच speculative-drafting approaches: native MTP, Gemma 4 MTP, EAGLE-3, DFlash, aur DSpark ka examine karenge. Ye methods target model se draft component ko information kaise milti hai aur candidate tokens ko sequential, autoregressive, parallel, ya hybrid approach se generate kya jata hai ismein differ karte hain. Akhir mein, ham hamare environment mein methods ko enable karna dikhayenge, hamare experiments se nikalne wale measurements report karenge jo AMD Instinct↓ MI300X aur MI355X GPU par ROCm↓ open software platform use karte hain, aur practical tuning aur observability considerations discuss karenge.

autoregressive decoding baseline: Standard autoregressive decoding mein, har decode step naya token produce karta aur commit karta hai. charanon ki example, charanon ki generation charanon sequential decode steps chahiye padte hain: Step 1:context→model→T1 Step 2:context + T1→model→T2 Step 3:context + T1 T2→model→T3 Step 4:context + T1 T2 T3→model→T4 Har step ke baad, generated token sequence mein append ho jata hai aur next step ka input ban jata hai. ye decoding loop seedha hai, lekin har output token ke liye ek model decode step zaroori hota hai. lambi generations ke is token-by-token loop latencity dominate kar sakti hai aur serving throughput limit kar sakti hai.

Key question behind speculative decoding ye hai: kya hum original model output behavior preserve karte hue, sirf ek token baar mein generation advance karne ki frequency kam kar sakte hain? Speculative decoding ye address karta hai proposal aur verification ko alag karke. Ek draft component pehle candidate future tokens propose karta hai. Original model, target model banke, uske baad unhe verify karta hai jab tak unhe commit nahi kareya jata.

Speculative decoding ka core idea: Speculative decoding original model replace nahi karta. Bas, original model target model rakhta hai jo final output ke liye responsible hai, aur iske aage faster proposal stage add karta hai. Is process ke do parts hain: Draft: candidate future tokens propose. Verify: target model check karta hai candidate.

Har ek speculative decoding round mein, jaise ki Figure 1 mein diwaliya gayi hai, ek lightweight draft component future tokens propose karta hai. Ye tokens sirf candidates hain aur unhe immediately commit nahi kareya jata. Target model phir ek verification pass mein candidate token sequence evaluate karta hai. Verification left to right proceed karta hai. Har draft token use target model ke corresponding position ke result se check kiya jata hai. Accepted tokens output sequence mein commit kiye jate hain. jab koi draft token reject hota hai, baad ke candidates...