---
title: '59894'
pageLayout: default
slug: thoughts/inference/vllm-prs/59894
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/59894.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#59894](https://github.com/aarnphm/aarnphm.github.io/issues/59894) \[Perf\]\[DSv4.1\] Trim the last KV-source layer in decoder replay

[vllm-project/vllm#59894](https://github.com/vllm-project/vllm/pull/59894) · gcanlin · MERGED · created 2026-10-03T18:34:59Z · merged 2026-10-09T06:07:50Z

## Purpose

In decoder replay steps, the last KV-source layer (layer 20 on DSV4.1-Flash) still ran on every row, though for the trimmed rows only its KV (SWA, compressed, index K) is read. Its indexer scoring and top-k, sparse MLA, wo and MoE feed only the replay layers.

- The cut layer now writes its KV for every row (`DeepseekV4DecoderLayer.write_kv` → `DeepseekV4Attention.forward_kv`), then runs as the first replay layer (`decoder_replay_start = cut`) on the replay rows, in the replay batch and its CUDA graphs. Its sliding-window metadata keeps the batch’s window starts, so it stays exact. The indexer row-buffer compaction goes.
- PIECEWISE graphs now decide the replay break (and `write_kv`) by graph size (`DecoderReplayLayers.replays()`) instead of `replay_batch is not None`, which is unset at capture: before, the break was never captured and PIECEWISE steps did not trim.

## Results

DSV4.1-Flash DEP4 on B200, CUDA graphs up to 8192 tokens, `vllm bench serve` random. Baseline: [#59532](https://github.com/aarnphm/aarnphm.github.io/issues/59532) with PIECEWISE trimming working, layer 20 not cut.

|                                     | baseline | this PR |       |
| ----------------------------------- | -------: | ------: | ----: |
| 8k in / 1 out, C16, total tok/s     |  236,952 | 245,556 | +3.6% |
| 32k in / 1 out, C16                 |  235,791 | 245,996 | +4.3% |
| 128k in / 1 out, C4                 |  194,012 | 204,764 | +5.5% |
| 32k in / 256 out, C32, output tok/s |    1,281 |   1,303 | +1.7% |

GSM8K 5-shot: 0.9606 both.

AgentX C128 (InferenceX DEP4 recipe, synthetic AL 3.51, 3,600 s), nightly [`43b4aae`](https://github.com/aarnphm/aarnphm.github.io/commit/43b4aaea) + this PR vs. the InferenceX CI run ([37070984585](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/37070984585), dev image ac9126):

|                            |                 CI |            this PR |       |
| -------------------------- | -----------------: | -----------------: | ----: |
| output tok/s               |              5,446 |              5,575 | +2.4% |
| TTFT mean / p50 / p99 (ms) |  293 / 163 / 2,558 |  255 / 155 / 2,004 |       |
| ITL mean / p50 / p99 (ms)  | 8.66 / 7.60 / 29.3 | 8.21 / 7.36 / 24.0 |       |
| per-user output tok/s      |              132.0 |              136.6 | +3.5% |
| requests completed         |             18,804 |             19,572 | +4.1% |

Prefix cache hit 95.2% in both.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---

## comments

**gcanlin** (2026-10-03T18:38:11Z):

@ivanium @Juntian777 Could you please take a look? When enabling decoder replay, we can cut some computation in 20th layer, like Sparse MLA and MoE.

**Juntian777** (2026-10-06T05:19:57Z):

@gcanlin Hi Canlin, thanks for the contribution! We just merged [#59532](https://github.com/aarnphm/aarnphm.github.io/issues/59532), which should further improve decoder replay performance. Would you mind rebasing this PR on top of the latest main, including [#59532](https://github.com/aarnphm/aarnphm.github.io/issues/59532), and rerunning the benchmarks with a larger CUDA graph size (e.g. 8192) so we can evaluate the incremental gains on top of it?

Also, while rebasing, could you please do a quick cleanup of the PR where applicable, removing any changes that are no longer needed or are now covered by [#59532](https://github.com/aarnphm/aarnphm.github.io/issues/59532), so that the remaining diff is easier to review? Thanks!

**mergify** (2026-10-06T05:40:15Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @gcanlin.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**mergify** (2026-10-08T04:18:28Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @gcanlin.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**WoosukKwon** (2026-10-09T04:07:55Z):

/ci run

**github-actions** (2026-10-09T04:09:55Z):

✅ Triggered [Buildkite CI #93763](https://buildkite.com/vllm/ci/builds/93763) for commit `afea5427a490`.

