---
title: '56227'
pageLayout: default
slug: thoughts/inference/vllm-prs/56227
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/56227.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#56227](https://github.com/aarnphm/aarnphm.github.io/issues/56227) \[Feat\]\[Model\] Support encoder-side SWA-bounded replay for DeepSeek-V4.1-Flash

[vllm-project/vllm#56227](https://github.com/vllm-project/vllm/pull/56227) · ivanium · MERGED · created 2026-09-10T07:44:40Z · merged 2026-09-18T22:01:24Z

## Purpose

DeepSeek-V4.1 keeps a 128-token sliding-window (SWA) KV cache per layer beside its prefix-cacheable compressed MLA and indexer caches. Storing that window for prefix caching and KV connectors costs more than it saves, so this implements DeepSeek’s encoder-side “SWA bounded replay”: the SWA caches leave prefix caching, and after a prefix hit ending at H the scheduler recomputes the hit’s last window \[H-128, H) to rebuild them. The replayed tokens keep their cached KV in every other group, their window attention reads nothing below H-128, and decode always sees a full window. Outputs after a hit are therefore not bit-identical to a cold run, by design.

- KV cache spec: `SlidingWindowMLASpec(bounded_replay=True)` derives `prefix_cacheable=False` and `prefix_replay_tokens=sliding_window`; `UniformTypeKVCacheSpecs` forwards both to the worker.
- Scheduler: a hit records `Request.replay_start = H-128` and rewinds the computed count by the window; async KV-connector loads replay once the KV arrives; chunks ending inside the replayed range allocate nothing new; hits no longer than the window are not adopted.
- Worker: only DeepSeek-V4.1 replays, so `DeepseekV41ModelState` owns the per-request replay start. `add_request` stores it; `prepare_attn` gathers it for the batch’s prefills, pads the replayed tokens’ slots in the prefix-cacheable groups with one Triton kernel, and hands the starts to the sparse SWA metadata builder through a `ModelSpecificAttnMetadata`. The runner core is untouched apart from an optional `model_specific_attn_metadata` argument on `DefaultModelState.prepare_attn`.
- DeepSeek-V4.1 kernels clamp the window at `replay_start`: the SWA index and gather-length kernels, the FlashInfer mixed index kernel and the FlashMLA combine kernel. The compressed slot mappings (MLA and indexer) inherit the padding of the tokens’ own slots, so a replayed token never re-enters the compressed caches.
- Switch: `--swa-bounded-replay` (`CacheConfig.swa_bounded_replay`, default on), read by the V4.1 attention layer. Needs model runner V2 and no prefill context parallelism; otherwise it logs once and keeps the SWA cache in prefix caching.
- One bugfix rides along as its own commit: KV-connector hits are taken in whole blocks under replay.

Known gap: a running request restarted in place under `kv_load_failure_policy=recompute` keeps its worker-side replay boundary and would recompute with a stale clamp; a follow-up will preempt such requests instead. No other open PR implements SWA replay.

## Usage

Nothing to configure: DeepSeek-V4.1 replays by default on model runner V2. `--no-swa-bounded-replay` restores the previous behaviour (SWA KV stored in the prefix cache). With replay on, KV connectors transfer only the MLA and indexer caches, and hits of at most 128 tokens are not adopted.

## Test Plan



```
pytest tests/v1/core/test_prefix_replay.py tests/v1/core/test_scheduler.py tests/v1/core/test_contiguous_kv_packing.py \
  tests/models/test_deepseek_v41_replay_start.py tests/v1/attention/test_deepseek_v4_swa_visible.py \
  tests/v1/attention/test_indexer_deepseek_v4_slot_mapping.py \
  tests/v1/kv_connector/unit/test_kv_load_failure_recovery.py
```

Accuracy: `deepseek-ai/DeepSeek-V4.1-Flash` on TP4 GB200 with `vllm serve`, gsm8k (5-shot) run twice against the same server; pass 2 re-sends every prompt, so each request is a full-prompt prefix hit that replays only its last window (92.6% prefix-cache hit rate).

## Test Result

All listed tests pass; pre-commit and mypy are clean.

| gsm8k, n=1319 (pass 1 cold / pass 2 all hits) | replay on (this PR) | replay off (`--no-swa-bounded-replay`) |
| --------------------------------------------- | ------------------- | -------------------------------------- |
| exact_match (flexible-extract)                | 0.9272 / 0.9212     | 0.9295 / 0.9295                        |

Both arms are within run-to-run noise (one standard error at n=1319 is 0.0075); the gap is the same in pass 1, where only the shared few-shot prefix is replayed. An earlier two-pass GPQA-diamond run on an internal pre-release checkpoint showed the same parity (0.6869 / 0.6919 with replay vs 0.7020 / 0.6869 without, n=198).

🤖 Developed with [Claude Code](https://claude.com/claude-code); all changes reviewed and tested by the author.

---

<details>
<summary> Essential Elements of an Effective PR Description Checklist </summary>

- [x] The purpose of the PR, such as “Fix some issue (link existing issues this PR will resolve)”.
- [x] The test plan, such as providing test command.
- [x] The test results, such as pasting the results comparison before and after, or e2e results
- [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.

</details>

---

## comments

**mergify** (2026-09-11T01:47:20Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @ivanium.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**mergify** (2026-09-13T01:10:38Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @ivanium.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**mergify** (2026-09-15T03:49:23Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @ivanium.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**ZJY0516** (2026-09-16T14:53:31Z):

There is a bug found by codex

There appears to be an off-by-one mismatch between SWA block allocation and bounded replay when the cache-hit length is not block-aligned, for example after an external KV-cache hit.

Consider:

- `window_size = 128`
- `block_size = 128`
- a hit covering 255 tokens, i.e. positions `[0, 255)`

The existing SWA allocation logic computes:



```python
num_skipped_tokens = H - window_size + 1  # 128
num_skipped_blocks = num_skipped_tokens // block_size  # 1
```

It therefore replaces logical block `0`, covering positions `0–127`, with the shared null block.

This is correct for ordinary continuation at position `255`, which only needs the previous `127` tokens plus the current token.

However, bounded replay starts one token earlier:



```python
replay_start = H - window_size  # 127
```

Position `127` must now be recomputed, but its block-table entry points to the null block.

**ivanium** (2026-09-16T22:26:41Z):

> There is a bug found by codex
>
> There appears to be an off-by-one mismatch between SWA block allocation and bounded replay when the cache-hit length is not block-aligned, for example after an external KV-cache hit.
>
> Consider:
>
> - `window_size = 128`
> - `block_size = 128`
> - a hit covering 255 tokens, i.e. positions `[0, 255)`
>
> The existing SWA allocation logic computes:
>
>
>
> ```python
> num_skipped_tokens = H - window_size + 1  # 128
> num_skipped_blocks = num_skipped_tokens // block_size  # 1
> ```
>
> It therefore replaces logical block `0`, covering positions `0–127`, with the shared null block.
>
> This is correct for ordinary continuation at position `255`, which only needs the previous `127` tokens plus the current token.
>
> However, bounded replay starts one token earlier:
>
>
>
> ```python
> replay_start = H - window_size  # 127
> ```
>
> Position `127` must now be recomputed, but its block-table entry points to the null block.

Technically, this should not happen as prefix cache hit should be block-size aligned, and in this case it is 128 tokens. However, one edge case allows KV offloading connectors to optionally return a non-block-aligned cache-hit length, which we don’t support anyway given the compressor circular buffer. I can enhance that API in this PR or maybe another one. Let me give it a try

**mergify** (2026-09-18T02:50:36Z):

Hi @ivanium, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

**ivanium** (2026-09-18T04:49:35Z):

/ci run

**github-actions** (2026-09-18T04:49:53Z):

✅ Triggered [Buildkite CI #89780](https://buildkite.com/vllm/ci/builds/89780) for commit `a210a63f184f`.

**ivanium** (2026-09-18T07:24:31Z):

/ci run

**github-actions** (2026-09-18T07:24:49Z):

❌ This PR is 3 commits behind upstream `main`. Your branch must contain every commit currently on upstream `main`. No new CI build was started. Merge or rebase onto the latest `main`, then rerun `/ci run`. To test this branch at your own risk, use `/ci run --allow-stale`.

<!-- vllm-ci-command:5726667574 -->

**WoosukKwon** (2026-09-18T08:51:09Z):

/ci run

**github-actions** (2026-09-18T08:53:28Z):

✅ Triggered [Buildkite CI #89810](https://buildkite.com/vllm/ci/builds/89810) for commit `4fad82b9b40a`.

**ivanium** (2026-09-18T17:40:04Z):

/ci run

**github-actions** (2026-09-18T17:40:23Z):

❌ This PR is 26 commits behind upstream `main`. Your branch must contain every commit currently on upstream `main`. No new CI build was started. Merge or rebase onto the latest `main`, then rerun `/ci run`. To test this branch at your own risk, use `/ci run --allow-stale`.

<!-- vllm-ci-command:5733860995 -->

**ivanium** (2026-09-18T18:44:03Z):

/ci run

**github-actions** (2026-09-18T18:44:24Z):

✅ Triggered [Buildkite CI #89895](https://buildkite.com/vllm/ci/builds/89895) for commit `4909bfff0ed7`.

**ChuanLi1101** (2026-09-21T08:38:54Z):

@ivanium @WoosukKwon Follow-up: on ROCm this default-on path faults after a prefix hit that spans a prefill chunk (`_pad_replayed_slots_kernel` / HSA memory fault). The worker clamp is FlashInfer/FlashMLA-only; amd/ only forwards `replay_start`.

ROCm is guarded in [vllm-project/vllm#57906](https://github.com/vllm-project/vllm/pull/57906) (SWA back in prefix cache). Remaining work is the AITER window clamp and a gated AMD `replay_start` test so this cannot land default-on without an AMD stamp.

