---
title: '61012'
pageLayout: default
slug: thoughts/inference/vllm-prs/61012
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/61012.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#61012](https://github.com/aarnphm/aarnphm.github.io/issues/61012) \[Bugfix\]\[Mooncake\] Use the EAGLE replay boundary when retaining sliding-window tails

[vllm-project/vllm#61012](https://github.com/vllm-project/vllm/pull/61012) · stu-cao · MERGED · created 2026-10-10T16:28:08Z · merged 2026-10-11T09:15:19Z

## Purpose

`MooncakeStoreCoordinator.store_mask` mirrors the core’s sparse sliding-window retention (`SlidingWindowManager.reachable_block_mask`) so the store keeps the same tails the local prefix cache does. It passed the non-EAGLE replay boundary, `(num_prompt_tokens - 1,)`, while masking with `use_eagle=True`. Under EAGLE the lookup matches one block past the hit and drops back from it, so the core’s `replay_boundaries` places the boundary one scheduler block lower; the store did not. For a prompt that is not block-aligned the retained tail therefore ended one block too high, the saved prefix held `need - 1` of the `cdiv(sliding_window - 1, block_size) + 1` contiguous blocks the lookup requires, and the joint hit collapsed to 0. Only block-aligned prompts hit.

Affected layout: a full-attention target group plus a sliding-window EAGLE drafter group (DFlash-style, e.g. window 2048 / block 64) with `prefix_cache_retention_interval=0`. Same family as [#57610](https://github.com/aarnphm/aarnphm.github.io/issues/57610) (sparse sliding-window retention keeping a tail exactly `need` blocks long), but on the store side and with a different cause; [#57613](https://github.com/aarnphm/aarnphm.github.io/issues/57613) composes with this change.

The fix extracts the core rule into a module-level `replay_boundaries()` in `vllm/v1/core/kv_cache_coordinator.py` (called directly from `KVCacheCoordinator.cache_blocks`, behaviour unchanged) and calls it from `_reachable_masks`, so the two sides cannot drift again. The non-EAGLE path still yields `(num_prompt_tokens - 1,)`. On a block-aligned prompt the store now also keeps the core’s lower “resend” tail: one extra sliding-window tail per aligned prompt, identical to what the local cache retains.

Not a duplicate: [#56615](https://github.com/aarnphm/aarnphm.github.io/issues/56615) pairs Mamba and attention boundaries for partial-tail handoffs and does not touch the retention boundary; [#57444](https://github.com/aarnphm/aarnphm.github.io/issues/57444) changes the core boundary alignment for fine-grained hits (same function, orthogonal change; trivial rebase whichever lands first); [#46906](https://github.com/aarnphm/aarnphm.github.io/issues/46906) sidesteps by making store retention dense; [#57613](https://github.com/aarnphm/aarnphm.github.io/issues/57613) fixes [#57610](https://github.com/aarnphm/aarnphm.github.io/issues/57610) inside `reachable_block_mask`.

## Test Plan

- `tests/v1/kv_connector/unit/test_mooncake_store_coordinator.py`: `test_store_mask_retention_interval_zero_eagle_keeps_lookup_tail` (the mask keeps the `need`-block tail at the lowered boundary, for an unaligned and an aligned prompt) and `test_store_mask_retention_interval_zero_eagle_round_trip` (save through `store_mask`, then the EAGLE lookup hits).
- Existing Mooncake coordinator, worker and scheduler unit tests, and `tests/v1/core/test_prefix_caching.py` + `tests/v1/core/prefix_cache` for the core refactor.

## Test Result

- Direct-call cleanup validation: `.venv/bin/python -m pytest tests/v1/kv_connector/unit/test_mooncake_store_coordinator.py tests/v1/core/test_prefix_caching.py -q -k "mooncake or retention or eagle or replay"` — 62 passed, 147 deselected. All pre-commit hooks passed on the three changed files. This cleanup used OpenAI Codex assistance.

- On B300: 677 passed across the suites above. The two `test_hybrid_prefix_cache_hit_rate` cases error identically on `main` in that environment (the test server cannot fetch its Hub model there) and are unrelated.

- On `main` the new tests fail for the unaligned prompt (mask mismatch; hit 0); with this change they pass.

- Lint: ruff check / format and mypy pass.

- Model output is unaffected: retention only decides which already-computed blocks the store keeps.

AI assistance: this change was developed with Claude (Anthropic) under human direction; every changed line was reviewed by the submitter, who ran the tests above.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---

## comments

**github-actions** (2026-10-11T07:39:54Z):

✅ @stu-cao, CI is now available for this PR.

- `/ci run` starts upstream CI; `/amd-ci run` starts AMD CI only.
- Your branch must contain every commit currently on its upstream target branch. Merge or rebase onto the latest target branch, then rerun the command. Append `--allow-stale` to a run command to test an outdated branch at your own risk.
- `/ci retry` retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
- `/amd-ci retry` retries failed jobs in AMD CI for the current PR head. Use `/amd-ci run` when the current head has no AMD CI build.
- `/ci cancel` cancels scheduled or running CI builds for this PR branch; `/amd-ci cancel` does the same for AMD CI only.

<!-- vllm-ci-authorized -->

**ivanium** (2026-10-11T07:57:12Z):

/ci run

**github-actions** (2026-10-11T07:57:31Z):

✅ Triggered [Buildkite CI #94067](https://buildkite.com/vllm/ci/builds/94067) for commit `2d0768c7b822`.

