Intuition: - we generate a small set of lookahead tokens, albeit 2-5 tokens with smaller speculators - uses the larger [[thoughts/Transformers#model]] to “verify” the input sequences + draft tokens (then replace tokens that aren’t valid from rejection sampler) In a sense, we are verify these in parallel instead of [[thoughts/Autoregressive models|autoregressive decoding]]. Given that verification is a lot cheaper comparing to next-token prediction A few techniques such as [[#ngrams|ngrams]], [[#EAGLE|EAGLE]] are supported in [[thoughts/vllm|vLLM]] ## papers - [Utility-Driven Speculative Decoding for Mixture-of-Experts](https://arxiv.org/abs/2506.20675) \[@saxena2025utilitydrivenspeculativedecodingmixtureofexperts\] ## EAGLE _Extrapolation Algorithm for Greater Language-model Efficiency_ - [EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test](https://arxiv.org/abs/2503.01840) \[@li2025eagle3scalinginferenceacceleration\] - [EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees](https://arxiv.org/abs/2406.16858) \[@li2024eagle2fasterinferencelanguage\] - [EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty](https://arxiv.org/abs/2401.15077) \[@li2025eaglespeculativesamplingrequires\] Motivation: - Some tokens are easier to generate than other tokens - Think of conjunctive adverbs, comparing to more complex word such as “annoyingly” - [[#speculative sampling|speculative sampling]] relies on the draft models having similar distributions as the target models. - use smaller models. i.e: Llama 3.2 3B as draft for Llama 3.3 70B. - high overhead for stepping through the whole models would outweighs the benefits > \[!note\] Difference between [EAGLE-1](#eagle-1) and [EAGLE-3](#eagle-3) > > - EAGLE-1’s limitation at its feature prediction constraints, via LM head architecture > - EAGLE-3 addresses this by use direct token prediction and rely on multi-layer feature fusion called “training-time test”, similar to [[#MLP Speculator|MLP Speculator]] > \[!tip\] distribution skew > > EAGLE _does not_ involve any fine-tuning of the target model, therefore preservation of outputs distributions by EAGLE is theoretically guaranteed for both greedy and non-greedy sampling. This is not the case with Lookahead and Medusa. ### EAGLE-1 Observations: > autoregressive on feature-level [^denote] is simpler than token-level, given that there are more regularity. [^denote]: features here refer to the hidden states of the decoder layers second-to-top-layer of the LLM, before the LM head. Not to be confused with [[thoughts/mechanistic interpretability#features|interpretable features]] > uncertainty in sampling process hinders the performance of predicting the next feature. > > _feature-level_ are high-dimensional and continuous, meaning sampling “am” or “always” will results in different feature sequences. EAGLE address this by **inputs the token sequence from one time step ahead including the sampling outcomes into the draft models**. - predicting $f_{\text{always}}$ based on $f_{\text{I}}$ and $t_\text{always}$ - predicting $f_{\text{am}}$ based on $f_{\text{I}}$ and $t_\text{am}$ ![[thoughts/images/eagle-feature-prediction-one-time-step.webp]] #### notation. - “Features” refers to second-to-top-layer feature of LLM, or the hidden states before LM head - Token by $t$, embedding by $e$, features by $f$, distributions by $p$ - Sequences are referred as $T_{i:j}$ for $(t_i, t_{i+1},\ldots, t_j)$ [^forward-pass-simplified] [^forward-pass-simplified]: Vanilla [[thoughts/Autoregressive models|autoregressive]] at token-level is described by $T_{1:j} \rightarrow E_{1:j} \rightarrow f_j \rightarrow p_{j+1} \rightarrow t_{j+1}$: - input $T_{1:j}$ is then transformed into embeddings $E_{1:j}$ - then into features $F_{1:j}$, - LM Head maps $f_j$ to a distribution $p_{j+1} = \text{LM\_Head}(f_j)$ - sampling next token $t_{j+1}$ #### architecture ![[thoughts/images/eagle-figure-5-comparison.webp]] ![[thoughts/images/eagle-figure-6-architecture.webp]] - See [vllm-project/vllm#20078](https://github.com/vllm-project/vllm/pull/20078) for triton fused ops. ```python [feature_seq, token_seq] # [bs, seq_len, hidden_dim], [bs, seq_len] token_seq -> token_emb # [bs, seq_len] -> [bs, seq_len, hidden_dim] fused_seq = feature_seq * token_emb # [bs, seq_len, 2 x hidden_dim] ``` - autoregressive\_head: - FC layer → `reduce # [bs, seq_len, hidden_dim]{:prolog}` - decoder layer → `features` - using [[thoughts/Attention#TreeAttention|tree attention]] to generate a draft tree of depth $m$ and more than $m$ tokens for $m$ forward pass. [^tree-attention] [^tree-attention]: Aligns with DistillSpec and Medusa #### training - Smooth [[thoughts/university/twenty-four-twenty-five/sfwr-4ml3/Bias and intercept#overfitting.|L1]] loss: $$ L_\text{reg} = \text{Smooth L1}(f_{i+1} \text{draft}(T_{2:i+1}, F_{1:i})) $$ - classification loss to optimize given objectives: $$ \begin{aligned} p_{i+2} &= \text{Softmax}(\text{LM\_Head}(f_{i+1})) \\ \hat{p}_{i+2} &= \text{Softmax}(\text{LM\_Head}(\hat{f}_{i+1})) \\ L_{\text{cls}} &= \text{CrossEntropy}(p_{i+2}, \hat{p}_{i+2}) \end{aligned} $$ - Autoregressive head with loss $L = L_{\text{reg}} + w_{\text{cls}} L_{\text{cls}}$ - set $w_{\text{cls}}=0.1$ given that classification loss is in order magnitude bigger than regression loss - Dataset: ShareGPT, 68k dialogue - Hyperparameter: - LR: $3e^{-5}$ - AdamW with beta $(\beta_1, \beta_2)=(0.9,0.95)$ - gradient clipping: $0.5$ ### EAGLE-2 tl/dr: Improvement on EAGLE-1 via context-aware dynamic draft tree into this drafting modeling. ### EAGLE-3 ![[thoughts/images/eagle-3-inference-pipeline.webp]] > \[!note\] Qwen3 replication > >Speculative execution for LLMs is an excellent inference-time optimization.
— Andrej Karpathy (@karpathy) 31 août 2023
It hinges on the following unintuitive observation: forwarding an LLM on a single input token takes about as much time as forwarding an LLM on K input tokens in a batch (for larger K than you might… https://t.co/FiwTwqsfho
Algorithm 1 SpeculativeDecodingStep
Input:
Sample guesses from
for to do
end for
Run in parallel
Determine the number of accepted guesses
Adjust ’s distribution if needed
if then
end if
Emit one token from and from
return