---
title: '54713'
pageLayout: default
slug: thoughts/inference/vllm-prs/54713
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/54713.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#54713](https://github.com/aarnphm/aarnphm.github.io/issues/54713) \[BugFix\] Retain both replay boundaries so an EAGLE resend of a block-aligned prompt still hits

[vllm-project/vllm#54713](https://github.com/vllm-project/vllm/pull/54713) · tobymao · MERGED · created 2026-09-01T06:25:25Z · merged 2026-09-10T08:39:25Z

## Purpose

Follow-up to [#53945](https://github.com/aarnphm/aarnphm.github.io/issues/53945), which fixed Mamba/SWA sparse-retention reuse under EAGLE/MTP by moving the replay boundary to where a lookup lands after the EAGLE drop:



```python
aligned = (
  request.num_prompt_tokens // scheduler_block_size * scheduler_block_size
)
return max(aligned - scheduler_block_size, 0)
```

That is the right position for a sibling whose prompt merely _starts_ with this one. It is not the right position for a resend of the **identical** prompt: `get_computed_blocks` caps a lookup at `num_tokens - 1`, because the last token has to be recomputed to obtain logits. So an identical resend matches one block lower and drops from there.

The two coincide unless the prompt length is an exact multiple of the scheduler block size — there they differ by exactly one block:

| prompt (block 32) | identical resend lands on | longer sibling lands on | [#53945](https://github.com/aarnphm/aarnphm.github.io/issues/53945) retains |
| ----------------- | ------------------------- | ----------------------- | --------------------------------------------------------------------------- |
| 127               | 64                        | 64                      | 64 ✅                                                                        |
| **128**           | **64**                    | 96                      | 96 ❌                                                                        |
| **160**           | **96**                    | 128                     | 128 ❌                                                                       |

With retention keeping only the higher position, every retained state sits above every candidate the resend’s lookup can produce, and the reconciled hit collapses to **0** — the same zero-hit failure sparse retention already avoids at unaligned prompt lengths.

## Fix

`get_replay_boundary` becomes `get_replay_boundaries` and returns both positions; `cache_blocks` retains each. Both collapse to a single position whenever the prompt is not block-aligned, so this adds at most one retained state per request.

Because it only ever _adds_ a reachable position, hit length cannot regress. Swept over 136 configurations (prompt lengths 128–392, identical resend and longer sibling, hash block size 16 and 64): **8 cases go from a 0-token hit to a real hit, 0 cases get shorter.**

## On the alignment

The boundary stays computed at `scheduler_block_size`, not the finer `_cache_hit_alignment_tokens`, deliberately. Fine-grained hits extend into the first non-full block only when a partial tail was registered there, so a hash-granular boundary over-estimates how far a lookup reaches and names a position _above_ the real candidate. Measured: doing that regresses prompt lengths 160/176/224/240 from a real hit to 0.

## Test Plan

`tests/v1/core/test_prefix_caching.py::test_hybrid_mamba_retention_mtp_resend_of_aligned_prompt` — a 128-token prompt (exact multiple of the 32-token alignment) with a Mamba “align” MTP group at `retention_interval=0`, asserting the resend hit and the longer sibling’s hit.

## Test Result

Fails on `main` with `assert 0 == 64` — the resend gets no cache hit at all. Passes with this change. `tests/v1/core` shows no new failures.

## Related, not fixed here

`OffloadingConnectorScheduler` builds its own `reachable_boundaries` from `req.num_prompt_tokens - 1` and never applies the EAGLE drop at all, feeding the same `reachable_block_mask`. That looks like the same class of bug at a third site, but the offloading path deliberately treats “no annotated eagle group” as non-EAGLE ([#52735](https://github.com/aarnphm/aarnphm.github.io/issues/52735)), so its model differs by design and it deserves its own change rather than a drive-by here.

---

## comments

**tobymao** (2026-09-01T06:54:51Z):

Updated after review: two defects in the first cut, both now pinned by tests.

**The back-off is one alignment unit, not one block.** The full-attention finder subtracts `min(alignment_tokens, its block_size)` then re-floors to the alignment, landing exactly one alignment unit below the boundary either way. Backing off one _Mamba_ block is only correct when the two are equal; when the groups’ block sizes differ (alignment = their LCM) the retained state sits where Mamba’s own finder rejects it, so the hit stays 0 and the extra block is dead weight. Now computed in tokens by `reachable_hit_positions()`, exact for alignment greater than, equal to, and less than the block size.

**Retention must key off “any group drops”, not “this group is eagle”.** The coordinator reconciles every group to one hit length, so a drop anywhere shortens the candidate offered here. The first cut read the group’s own `use_eagle` bit and worked only through the coordinator’s flag-all fallback (no annotator exists for `glm5_next`); the day one lands, the zero-hit bug returns silently. The coordinator now sets `lookup_drops_eagle_block` on every manager from `bool(self.eagle_group_ids)`.

Tests: the MTP test is parametrized over all three annotation routes (flag-all fallback, full-attention-only, both) and now runs with real speculative blocks and lookahead; a new differing-block-size test pins the alignment-unit back-off. Both fail on the previous cut (`full_only`: “mamba hash 1 should be cached”; back-off: “reachable state missing; cached=\[5\]”). 95/95 in `test_prefix_caching.py`.

Two related defects found in review and deliberately **not** fixed here, since each is pre-existing and independent:

- `SlidingWindowManager.reachable_block_mask` has the same shape of bug on this path: its `shift=+1` retention keeps a run the post-drop lookup cannot see, so an SWA group reconciles to 0 exactly as Mamba did whenever the full-attention group also drops.
- Section (2) of the SWA and Mamba masks are now near-duplicates; a shared helper on the base class would give a third sparse-retention spec the behavior for free. Happy to fold either in here if preferred.

**kamb-code** (2026-09-03T06:20:42Z):

I am rebuilding [#53479](https://github.com/aarnphm/aarnphm.github.io/issues/53479) as a focused scheduler follow-up to this PR rather
than duplicating its reachable-state logic. The local composition passes the
focused CPU matrix.

GitHub currently reports DCO `action_required`, and neither public commit
(`5ad9ac6` or `d3bdc5f`) contains a `Signed-off-by` trailer. Because that
certification needs to come from the original contributor, could you add the
sign-offs when convenient? I will wait for this PR to merge, or reassess if
it closes unmerged, rather than rewriting or signing over your commits.

AI assistance from Claude Code and OpenAI Codex was used in the composition
review; I reviewed and verified this note.

**khluu** (2026-09-06T08:22:04Z):

/ci run

**github-actions** (2026-09-06T08:22:22Z):

✅ Triggered [Buildkite CI #87430](https://buildkite.com/vllm/ci/builds/87430) for commit `d3bdc5f7cc60`.

**twu3202** (2026-09-06T11:47:39Z):

Independent data point on a different hybrid family: Qwen3.6-35B-A3B W4A16 (GDN + full-attention MoE, 3 Mamba groups + 1 full-attention group, `align`), MTP k=1, synthetic prompts, single run, `main` @ [`8a0a7ee`](https://github.com/aarnphm/aarnphm.github.io/commit/8a0a7ee) with this PR applied.

| block                                    | arm                                    | probe hits / 72,992 | 1st repeat, 1,308-tok prefix | 1st repeat, 6,008-tok prefix |
| ---------------------------------------- | -------------------------------------- | ------------------- | ---------------------------- | ---------------------------- |
| 1056 (default)                           | main                                   | 0                   | 0                            | 0 (2nd repeat: 4,224)        |
| 1056 (default)                           | + this PR                              | 0                   | 0                            | **4,224**, TTFT 396 → 110 ms |
| 1056 (default)                           | + this PR + `disable_eagle_block_drop` | 22,176              | 1,056                        | not run                      |
| 544 (`--mamba-ssm-cache-dtype bfloat16`) | main                                   | —                   | 0                            | —                            |
| 544 (`--mamba-ssm-cache-dtype bfloat16`) | + this PR                              | —                   | **544**, TTFT 103 → 54 ms    | —                            |

Block size equals the prefix-match alignment here, so the extra retained state sits at `aligned − block`: a shared prefix that floors to one block still lands on 0, and the PR starts paying at two blocks — 2,112 tokens by default, 1,088 with the bf16 SSM state, where a persona-sized 1,308-token prefix hits on the first repeat. Unpatched main reaches the same steady state one request later, so the win here is one saved cold request per prefix. Same picture at k=3 (block grows to 1072, hits 2,144 / 4,288). Stacking with `disable_eagle_block_drop` is clean (22,176, identical to the no-speculation baseline), and acceptance is unchanged by either change (57.9–60.2% across the k=1 arms).

DCO is still `action_required`, so the commits need a `Signed-off-by` line before this can merge.

(Runs assisted by Claude; reviewed by me.)

**ZJY0516** (2026-09-07T03:09:50Z):

@tobymao Does `disable_eagle_block_drop ` can solve your problem?

**tobymao** (2026-09-07T03:21:36Z):

Not really, it works by turning off trailing-block drop entirely (for every group), which removes the acceptance-rate protection that drop exists for, whereas my fix keeps that protection and just corrects the retention math.

**mergify** (2026-09-08T14:42:56Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @tobymao.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**Suppressor72** (2026-09-08T15:52:52Z):

Tested the current patch on the layout from [#53504](https://github.com/aarnphm/aarnphm.github.io/issues/53504): Qwen3.8-27B-FP8,
hybrid Mamba/GDN in `align` mode, in-checkpoint MTP K=2, FP8 KV, TP=2.
The base was stock nightly `54da70c1e`; the workload was a serialized cold
population followed by three identical repeats of a 4,481-token prompt. Each
arm used one fresh engine boot (`n=1` per arm).

The vLLM stats line reports a cumulative hit rate, so I converted its sequence
to the incremental cached-token count for each request:

| config                                                                                                      | repeat 1 | repeat 2 | repeat 3 |
| ----------------------------------------------------------------------------------------------------------- | -------: | -------: | -------: |
| stock default                                                                                               |        0 |    1,600 |    1,600 |
| + this PR                                                                                                   |    1,600 |    1,600 |    1,600 |
| flag only (`disable_eagle_block_drop`, [#53388](https://github.com/aarnphm/aarnphm.github.io/issues/53388)) |    3,200 |    3,200 |    3,200 |
| + this PR + flag                                                                                            |    3,200 |    3,200 |    3,200 |
| no speculation                                                                                              |    3,136 |    3,136 |    3,136 |

MTP resolved a 1,600-token alignment/block size, while the no-speculation
control resolved 1,568, hence 3,200 versus 3,136 at the same two-full-block
ceiling.

This gives me three useful checks:

1. Under the default drop, this PR turns the first-repeat total miss into a
   one-alignment-unit hit and keeps doing so on later repeats. The apparent
   `17.9% / 23.8% / 26.8%` progression in the raw log is only accumulation
   across the cold request and repeats, not progressively increasing reuse.
2. This PR plus `disable_eagle_block_drop` matches the flag-only arm exactly:
   no composition regression on this layout. Our earlier local fix retained
   the shifted state _instead of_ the boundary state and therefore lost this
   property; retaining both positions here is the correct formulation.
3. A positive retention interval of 1,600 produced the same hit counts on this
   prompt. That policy retains a state at every 1,600-token segment boundary,
   whereas this PR preserves sparse-retention behavior by adding only the
   shifted reachable position for each retained boundary. The one-block
   difference between this PR’s default arm and the flag-on arm is exactly
   consistent with the configured EAGLE back-off on this geometry; none of
   the tested retention policies changed that lookup cap.

The newly merged v0.29 release changes make dense retention the unset default
for hybrid+EAGLE configurations ([#55760](https://github.com/aarnphm/aarnphm.github.io/issues/55760)/[#55861](https://github.com/aarnphm/aarnphm.github.io/issues/55861)). These measurements are about
the still-distinct sparse-retention path (`prefix_cache_retention_interval=0`),
where this patch avoids the first-repeat zero hit without disabling the EAGLE
drop.

On this layout, the patch behaves as intended and composes cleanly with the
opt-out flag.

_Measurements and draft prepared with agent assistance._

**tobymao** (2026-09-08T20:49:59Z):

Rebased onto `main`, and the PR has been repointed — the description above is rewritten to match.

[#53945](https://github.com/aarnphm/aarnphm.github.io/issues/53945) landed while this was open and fixes the failure this PR originally targeted, by moving the replay boundary down to the post-drop position instead of retaining an extra state below it. That approach is better than what this PR had, so the old mechanism (`reachable_hit_positions`, `lookup_drops_eagle_block`) is gone entirely.

What survives is a narrower bug in that new boundary, found while rebasing. `get_replay_boundary` computes from `floor(num_prompt_tokens / block)`, but `get_computed_blocks` caps a lookup at `num_tokens - 1` so the last token can be recomputed for logits. A resend of the identical prompt therefore matches one block lower than a sibling whose prompt merely starts with this one, and drops from there. The two positions coincide except when the prompt length is an exact multiple of the scheduler block size — and there, retention keeps only the higher one, so the resend’s every retained state sits above every candidate its lookup can produce and the reconciled hit collapses to 0.

The new test fails on current `main` with `assert 0 == 64` — a total miss on an identical resend of a 128-token prompt.

The fix returns both positions. Since it only ever adds a reachable position, hit length cannot regress; swept over 136 configurations, 8 go from a 0-token hit to a real hit and none get shorter.

I also tried computing the boundary at the fine-grained hash alignment rather than `scheduler_block_size`, on the theory that the drop is `min(alignment_tokens, block_size)`. That is wrong and I dropped it: phase-1 matching is bounded by the full-attention block size, and phase 2 only extends into the first non-full block when a partial tail was registered there, so a hash-granular boundary over-estimates the reach. Measured, it regresses lengths 160/176/224/240 from a real hit to 0.

**ZJY0516** (2026-09-10T01:37:24Z):

/ci run

**github-actions** (2026-09-10T01:37:42Z):

✅ Triggered [Buildkite CI #88018](https://buildkite.com/vllm/ci/builds/88018) for commit `1d8afd3802bf`.

**github-actions** (2026-09-10T01:37:48Z):

✅ @tobymao, CI is now available for this PR.

- `/ci run` starts upstream CI; `/amd-ci run` starts AMD CI only.
- `/ci retry` retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
- `/amd-ci retry` retries failed jobs in AMD CI for the current PR head. Use `/amd-ci run` when the current head has no AMD CI build.
- `/ci cancel` cancels scheduled or running CI builds for this PR branch; `/amd-ci cancel` does the same for AMD CI only.

<!-- vllm-ci-authorized -->

