---
title: '58132'
pageLayout: default
slug: thoughts/inference/vllm-prs/58132
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/58132.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#58132](https://github.com/aarnphm/aarnphm.github.io/issues/58132) \[Model\] Decoder-side SWA bounded replay for DeepSeek-V4.1

[vllm-project/vllm#58132](https://github.com/vllm-project/vllm/pull/58132) · ivanium · MERGED · created 2026-09-22T10:07:47Z · merged 2026-09-29T08:25:47Z

## Purpose

Decoder-side SWA bounded replay for DeepSeek-V4.1, on top of the encoder side ([#56227](https://github.com/aarnphm/aarnphm.github.io/issues/56227)). The layers past the last KV-source layer (21-39 of 40 on V4.1-Flash) own only sliding-window KV, so in eager prefill steps they run on each request’s last 128 tokens, cold prefills included. Simplified re-submission of [#56752](https://github.com/aarnphm/aarnphm.github.io/issues/56752) (closed); no other open PR implements this.

- `DeepseekV41ModelState.prepare_attn` prepares the kept rows as a sub-batch, the way a microbatch is prepared: one Triton kernel gathers rows, positions, slot mappings and the replay-layer KV start, and the sub-batch gets attention metadata and a forward context of its own. DP ranks agree on whether any rank trims and on every rank’s replay token count.
- `DecoderReplayLayers` gathers the layer inputs by those rows, runs the replay layers under that context and scatters the outputs back.
- Only eager steps trim. A step that fits a CUDA graph (up to 1024 tokens by default) keeps the replay layers on its whole batch inside the graph: such steps are launch-bound, so the graph is worth more than the trim. `prepare_attn` already gets the step’s graph mode, which DP ranks share, so there are no nested captures, no eager breaks and no runner changes; the diff is confined to `vllm/models/deepseek_v41/`, a config docstring and the CI job list.
- With the fused MoE finalize of [#58586](https://github.com/aarnphm/aarnphm.github.io/issues/58586), a decode step’s cut layer hands its unfinalized `MoEOutput` to the replay layers unchanged, and layer 21’s first boundary finalizes it. A trimming step holds a prefill longer than the window, more tokens than any step that defers its finalize, so the trim path only sees tensors.
- Stays off, with a warning, when the PP stage does not hold every layer after the cut, under sequence-parallel MoE, PCP or microbatching, with an Engram layer after the cut, or with a drafter window wider than the target’s. `--no-swa-bounded-replay` turns off both sides. A no-op on ROCm.

## Test Plan



```
pytest tests/models/test_deepseek_v41_replay_batch.py tests/models/test_deepseek_v41_decoder_replay_layers.py
```

- Accuracy: gsm8k 5-shot, n=1319, two passes against one server (pass 2 is all full-prompt prefix hits).
- Performance: `vllm-bench`, random token-id prompts, prefix cache reset before every point.
- Both on V4.1-Flash, TP4 and DP4 x TP1 x EP4 (DEP4), decoder side on vs off with the encoder side on in both (off via a local patch, not in this PR).

## Test Result

Unit tests cover the replay rows and sub-batch metadata (adaptive verification, prompt-logprobs requests, DP agreement), graph steps getting no replay batch, and the gather/scatter with the indexer buffers. pre-commit and mypy are clean.

### Accuracy (GB200, V4.1-Flash)

| arm                      | pass 1 (cold) | pass 2 (prefix hits) |
| ------------------------ | ------------- | -------------------- |
| TP4, decoder replay on   | 0.9249        | 0.9234               |
| TP4, decoder replay off  | 0.9303        | 0.9249               |
| DEP4, decoder replay on  | 0.9333        | 0.9287               |
| DEP4, decoder replay off | 0.9257        | 0.9181               |

`exact_match/flexible-extract`; one standard error is 0.0072. 0 request errors. On TP4 the fused MoE finalize of [#58586](https://github.com/aarnphm/aarnphm.github.io/issues/58586) is active, so every decode step passes the cut layer’s unfinalized MoE output through the replay layers.

### Performance (GB200, V4.1-Flash)

Saturated prefill throughput with decoder replay stays at 83K-86K tok/s on TP4 and 199K-228K on DEP4 at every prompt length, against 68K-54K and 134K-120K without. From 16K up, prefill TTFT drops 25-36% on TP4 and 37-45% on DEP4 at any concurrency, and TPOT drops up to 29% and 39% at high concurrency.

TP4:

| ISL  | TTFT, 1 request | max conc | prefill tok/s at max conc | OSL 512 out tok/s at max conc | OSL 512 TPOT at max conc |
| ---- | --------------- | -------- | ------------------------- | ----------------------------- | ------------------------ |
| 1K   | 55 / 55 ms      | 256      | 83.1K / 68.0K (+22%)      | +6%                           | -2%                      |
| 4K   | 206 / 185 ms    | 256      | 82.8K / 65.5K (+26%)      | +7%                           | -7%                      |
| 8K   | 206 / 191 ms    | 256      | 86.0K / 64.9K (+33%)      | +20%                          | -16%                     |
| 16K  | 211 / 285 ms    | 256      | 85.8K / 63.6K (+35%)      | +25%                          | -20%                     |
| 32K  | 423 / 573 ms    | 128      | 86.1K / 62.2K (+38%)      | +29%                          | -22%                     |
| 64K  | 831 / 1178 ms   | 64       | 85.8K / 59.5K (+44%)      | +35%                          | -26%                     |
| 128K | 1667 / 2552 ms  | 32       | 85.5K / 54.4K (+57%)      | +40%                          | -29%                     |

DEP4:

| ISL  | TTFT, 1 request | max conc | prefill tok/s at max conc | OSL 512 out tok/s at max conc | OSL 512 TPOT at max conc |
| ---- | --------------- | -------- | ------------------------- | ----------------------------- | ------------------------ |
| 1K   | 209 / 197 ms    | 256      | 198.6K / 132.0K (+50%)    | +4%                           | -1%                      |
| 4K   | 212 / 200 ms    | 256      | 223.2K / 134.4K (+66%)    | +19%                          | -13%                     |
| 8K   | 215 / 213 ms    | 256      | 226.2K / 134.0K (+69%)    | +34%                          | -24%                     |
| 16K  | 240 / 383 ms    | 256      | 228.3K / 133.0K (+72%)    | +31%                          | -22%                     |
| 32K  | 479 / 765 ms    | 128      | 223.5K / 130.4K (+71%)    | +44%                          | -29%                     |
| 64K  | 945 / 1546 ms   | 64       | 221.2K / 126.4K (+75%)    | +38%                          | -26%                     |
| 128K | 1946 / 3217 ms  | 32       | 213.4K / 119.8K (+78%)    | +62%                          | -39%                     |

Decoder replay on / off. TTFT is the prefill-only p50. Each point is one run of 4 x concurrency prompts with the default `max_num_batched_tokens` 16384 (per rank on DEP4) and max CUDA graph size 1024; each arm runs on its own GB200 node, in parallel.

**Regressions.** Only prompts of 8K or less at concurrency 64 or less; the table has every such point on either setup, with decoder replay on vs off.

- A single 1K prompt runs the same code as the baseline, inside the CUDA graph: TP4 matches (55 / 55 ms); DEP4 measures within 1-6%. Trimming such steps, as the previous revision did, took TP4’s TTFT from 56 to 110 ms.
- Single 4K-8K prompts cost \~2.7 ms of CPU per eager step: 0.2 ms preparing the sub-batch, \~2 ms building its attention metadata with the same builders the main batch runs every step, 0.5 ms of gather/scatter. Those steps are launch-bound (\~175 ms of kernel launches with the GPU waiting on the CPU), so the trim saves no time there. With both TP4 arms on one node the gap is 0-3%; across nodes in the main runs it measures up to +21% on TP4 and +7% on DEP4, with throughput within 5%.
- The larger TTFT p50 swings at concurrency 4-64 change sign between points and setups, with throughput between -5% and +13%.

| ISL, conc | TP4 prefill-only TTFT p50 | TP4 OSL 512 TTFT p50 | TP4 OSL 512 out tok/s | DEP4 prefill-only TTFT p50 | DEP4 OSL 512 TTFT p50 | DEP4 OSL 512 out tok/s |
| --------- | ------------------------- | -------------------- | --------------------- | -------------------------- | --------------------- | ---------------------- |
| 1K, 1     | -1%                       | -2%                  | -1%                   | +6%                        | +1%                   | +0%                    |
| 1K, 4     | +5%                       | -33%                 | +2%                   | +5%                        | +9%                   | +4%                    |
| 1K, 16    | -12%                      | +11%                 | -1%                   | +5%                        | -3%                   | +6%                    |
| 1K, 64    | -18%                      | +11%                 | +9%                   | -13%                       | +8%                   | -4%                    |
| 4K, 1     | +11%                      | +21%                 | -5%                   | +6%                        | +7%                   | +0%                    |
| 4K, 4     | +6%                       | +14%                 | +4%                   | +12%                       | -20%                  | +5%                    |
| 4K, 16    | -21%                      | +23%                 | -2%                   | -13%                       | -0%                   | +13%                   |
| 4K, 64    | -22%                      | +5%                  | +7%                   | -41%                       | -38%                  | +7%                    |
| 8K, 1     | +8%                       | +19%                 | -5%                   | +1%                        | +7%                   | -0%                    |
| 8K, 4     | +9%                       | +22%                 | -5%                   | -22%                       | +18%                  | -0%                    |
| 8K, 16    | -23%                      | -18%                 | +2%                   | -36%                       | -25%                  | -5%                    |

Same-node check (TP4, both arms on one GB200 node, 10 single prefill-only requests each):

| ISL | TTFT p50 on / off         |
| --- | ------------------------- |
| 2K  | 188.6 / 183.1 ms (+3.0%)  |
| 4K  | 193.8 / 190.0 ms (+2.0%)  |
| 8K  | 197.7 / 198.3 ms (-0.3%)  |
| 16K | 207.8 / 285.8 ms (-27.3%) |

### AMD (MI355X, ROCm 7.2.3)

A no-op: the ROCm model classes have no `decoder_replay_layers`, and the encoder side stays off on ROCm as before. The new tests are not in the AMD mirror of the CI job. Checked at [`9a86c2c`](https://github.com/aarnphm/aarnphm.github.io/commit/9a86c2c99f) against its base [`496c647`](https://github.com/aarnphm/aarnphm.github.io/commit/496c6472cb) (the rebase onto main and the eager-only follow-up change only code ROCm does not run), V4.1-Flash TP4 with the recipes.vllm.ai MI355X command minus `--no-swa-bounded-replay` (DSpark, 5 speculative tokens):

|                            | this PR | base   |
| -------------------------- | ------- | ------ |
| gsm8k pass 1 (cold)        | 0.9030  | 0.8961 |
| gsm8k pass 2 (prefix hits) | 0.9045  | 0.8946 |

`tests/evals/gsm8k/gsm8k_eval.py` accuracy (5-shot, n=1319, concurrency 128), within noise. Prefix-sharing prompts of up to 256K tokens served without errors on both trees, with the same prefix-cache hits. The two new test files plus `test_deepseek_v41_replay_start.py` and `test_deepseek_v4_vl_rocm.py`: 23 passed.

🤖 Developed with [Claude Code](https://claude.com/claude-code); all changes reviewed and tested by the author.

---

## comments

**ivanium** (2026-09-25T23:44:04Z):

/ci run

**github-actions** (2026-09-25T23:44:22Z):

✅ Triggered [Buildkite CI #91349](https://buildkite.com/vllm/ci/builds/91349) for commit `ef562a74aa5c`.

**mergify** (2026-09-26T19:19:26Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @ivanium.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**ivanium** (2026-09-29T02:20:03Z):

/ci run

**github-actions** (2026-09-29T02:20:22Z):

✅ Triggered [Buildkite CI #91717](https://buildkite.com/vllm/ci/builds/91717) for commit `394e46fd7190`.

**ivanium** (2026-09-29T07:31:50Z):

/ci retry

**github-actions** (2026-09-29T07:33:00Z):

✅ Triggered [Buildkite CI #91784](https://buildkite.com/vllm/ci/builds/91784) for commit `e0f1dc431dd0`, running 1 failed step(s) from [Buildkite CI #91717](https://buildkite.com/vllm/ci/builds/91717).

