---
title: '59532'
pageLayout: default
slug: thoughts/inference/vllm-prs/59532
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/59532.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#59532](https://github.com/aarnphm/aarnphm.github.io/issues/59532) \[Perf\]\[DSv4.1\] Run decoder replay layers in CUDA graphs and trim in PIECEWISE graphs

[vllm-project/vllm#59532](https://github.com/vllm-project/vllm/pull/59532) · Juntian777 · MERGED · created 2026-10-01T03:09:16Z · merged 2026-10-06T05:11:45Z

## Purpose

Run the decoder-side SWA bounded replay layers ([#58132](https://github.com/aarnphm/aarnphm.github.io/issues/58132)) in CUDA graphs of their own, and trim them in PIECEWISE graph steps too. [#58132](https://github.com/aarnphm/aarnphm.github.io/issues/58132) trims layers 21-39 to each prefill’s last 128 tokens only on eager steps (above the model graph’s capture range) and runs them eagerly there; those steps are often launch-bound ([#58132](https://github.com/aarnphm/aarnphm.github.io/issues/58132) measured \~175 ms of kernel launches with the GPU waiting on a single 4K-8K prefill), and with a large `max_cudagraph_capture_size` nothing trims at all.

- `DecoderReplayCudaGraphManager` (`nvidia/decoder_replay_cudagraph.py`), a `CudaGraphManager` owned by `DeepseekV41ModelState`, captures breakable PIECEWISE graphs of the replay layers, keyed by the replay batch’s padded token count and reading the layer inputs from static buffers. Attention runs in their eager breaks: the replay batch mixes prefill tails and decodes, and the SWA and sparse MLA builders support uniform batches only.
- PIECEWISE model graphs of at least `compilation_config.decoder_replay_trim_threshold` padded tokens (768 by default) break out to the replay layers, which run on a replay batch (trimmed, or whole when nothing trims) and write their outputs to fixed buffers the graph’s next segment reads. FULL graphs and smaller PIECEWISE ones keep the layers on the whole batch, as before.
- `_prepare_replay_batch` pads the replay batch to the smallest graph that fits the largest DP rank’s replay batch and sets it on `DecoderReplayLayers` as one `ReplayBatch` (rows, forward context, graph runner). A replay batch above `max_cudagraph_capture_size`, possible only in eager steps, stays eager.
- Captures run the runtime path on a dummy batch, so the graphs read the buffers a real step’s replay batch fills; on MRV2 the SWA KV insert stays inside the graph and reads the slot mapping by address.
- `ModelState.capture_inner_cudagraphs` is a new hook the runner calls after the model’s capture; it is a no-op by default.
- On by default with breakable piecewise graphs (the default for DeepSeek-V4.1); `compilation_config.cudagraph_decoder_replay=false` turns it off. Also off under LoRA, prompt embeddings or on a non-first PP rank. Off keeps [#58132](https://github.com/aarnphm/aarnphm.github.io/issues/58132)’s behavior.

No open PR runs the replay layers in graphs. [#56752](https://github.com/aarnphm/aarnphm.github.io/issues/56752) (closed) captured them nested inside the model graph’s capture; this captures them on their own and has the model graph break out to them.

## Benchmark

Prefill-only TTFT of DeepSeek-V4.1-Flash on GB200 at `f21ea3c947` (the last commit only adds the switch and renames), default CUDA graph settings, `max_num_batched_tokens=16384`, prefix caching off, `max_tokens=1`. With the default capture range every 1K-128K prefill step is eager, so this measures the eager path, not PIECEWISE trimming. Median of 3 runs in ms; c1 is one request at a time, c8 is 8 concurrent. Off is `--no-swa-bounded-replay`; parent is [`df8fd42`](https://github.com/aarnphm/aarnphm.github.io/commit/df8fd42116) (main with [#58132](https://github.com/aarnphm/aarnphm.github.io/issues/58132)) with bounded replay on. Each topology ran all three on one node.

TP4 (repeats within ±2%; this PR’s 1K c1 runs were 102/103/131 ms):

| ISL  | conc |    Off | Parent |    This PR | vs off | vs parent |
| ---- | ---- | -----: | -----: | ---------: | -----: | --------: |
| 1K   | c1   |  160.9 |  165.9 |  **103.3** |  1.56x | **1.61x** |
| 1K   | c8   |  324.3 |  333.7 |  **269.1** |  1.21x | **1.24x** |
| 4K   | c1   |  165.4 |  172.5 |  **107.0** |  1.55x | **1.61x** |
| 4K   | c8   |  505.3 |  505.2 |  **442.7** |  1.14x | **1.14x** |
| 8K   | c1   |  170.0 |  177.8 |  **112.5** |  1.51x | **1.58x** |
| 8K   | c8   |  963.5 |  683.9 |  **562.9** |  1.71x | **1.21x** |
| 32K  | c1   |  540.1 |  376.0 |  **327.8** |  1.65x | **1.15x** |
| 32K  | c8   | 3984.8 | 2714.7 | **2292.4** |  1.74x | **1.18x** |
| 128K | c1   | 2267.1 | 1522.3 | **1392.1** |  1.63x | **1.09x** |
| 128K | c8   | 9847.5 | 6553.1 | **5933.2** |  1.66x | **1.10x** |

DEP2 (one fixed prompt per length; each ratio compares runs on the same GPU pair, since the two pairs of a node differ by 3-10%; c1 repeats within ±3%; c8 below 128K is omitted, it varies ±10-30% with how the two API servers split the 8 requests across DP ranks):

| ISL  | conc |    Off |    This PR | vs off | Parent |  This PR | vs parent |
| ---- | ---- | -----: | ---------: | -----: | -----: | -------: | --------: |
| 1K   | c1   |  103.8 |   **72.5** |  1.43x |  116.2 | **79.9** | **1.45x** |
| 4K   | c1   |  135.3 |   **92.3** |  1.47x |  117.4 | **94.7** | **1.24x** |
| 8K   | c1   |  246.9 |  **159.4** |  1.55x |  164.1 |    163.9 |     1.00x |
| 32K  | c1   |  983.1 |  **609.8** |  1.61x |  617.8 |    627.8 |     0.98x |
| 128K | c1   | 3950.5 | **2460.3** |  1.61x | 2518.0 |   2567.8 |     0.98x |
| 128K | c8   | 9114.6 | **5716.4** |  1.59x | 5467.9 |   5508.2 |     0.99x |

### Why TP4 gains at long prompts and DEP2 does not

A profile of one 128K prefill shows the trimmed layers need only a few ms of GPU per chunk, but on TP4 their eager launches take several times longer and layers 0-20 do not hide them, so the GPU sits idle through most of layers 21-39; in graphs that idle time is gone. A DEP2 rank runs layers 0-20 on the whole chunk without TP splitting, so the CPU is far ahead of the GPU and the eager launches were already hidden; this PR only helps DEP2 where the step is short (1K-4K).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---

## comments

**WoosukKwon** (2026-10-05T21:55:02Z):

/ci run

**github-actions** (2026-10-05T21:55:56Z):

✅ Triggered [Buildkite CI #92967](https://buildkite.com/vllm/ci/builds/92967) for commit `7c52deb6bad9`.

**vllm-agent** (2026-10-05T23:13:12Z):

<!-- ci-selector-shadow -->

### CI selector (shadow): 115 test steps (178 jobs) instead of 78 (100 jobs)

Shadow mode: this changes nothing about what CI runs. It shows what the evidence-based selector would pick for this PR, next to today’s rules. [How it works](https://github.com/vllm-project/ci-infra/tree/main/buildkite/ci_selector).

**Feedback welcome:** reply here if it would skip a step this change needs, or runs something unrelated.

| steps (jobs)           | Today’s rules | Selector  | Would skip | Would add |
| ---------------------- | ------------- | --------- | ---------- | --------- |
| NVIDIA, CPU and others | 78 (100)      | 115 (178) | 19 (23)    | 56 (101)  |
| AMD mirrors            | 72 (94)       | 97 (155)  | 17 (17)    | 42 (78)   |

<details><summary>Selector would run (115)</summary>

- `amd-kernels-mi355`
- `arm-cpu-test` ×3
- `async-engine-inputs-utils-worker`
- `basic-correctness` ×2
- `basic-correctness-cpu-offload`
- `basic-correctness-prefetch-offload`
- `basic-correctness-sleep-mode`
- `basic-models-tests-extra-initialization` ×14
- `basic-models-tests-initialization`
- `basic-models-tests-other`
- `batch-invariance-b200`
- `batch-invariance-h100`
- `benchmarks-cli-test`
- `cpu-kernel-tests` ×2
- `cpu-language-generation-and-pooling-model-tests` ×3
- `cpu-multi-modal-model-tests-n` ×4
- `cpu-multimodal-config`
- `cpu-params-env-tokenizers-parser`
- `cpu-reasoning-renderers`
- `cpu-spec-decode-tests`
- `cudagraph`
- `deepseek-v4-kernel-test-b200`
- `deepseek-v4-kernel-test-h100`
- `distributed-dp-tests-2-gpus`
- `distributed-dp-tests-4-gpus`
- `distributed-model-tests-2-gpus` ×3
- `distributed-tests-8xh100`
- `distributed-torchrun-examples-4-gpus`
- `distributed-torchrun-shutdown-tests-2-gpus`
- `e2e-core-1-gpu`
- `e2e-core-large-memory`
- `e2e-scheduling-1-gpu`
- `e2e-scheduling-accuracy-1-gpu`
- `elastic-ep-scaling-test`
- `engine`
- `engine-1-gpu`
- `entrypoints-integration-api-server` ×4
- `entrypoints-integration-llm`
- `entrypoints-integration-pooling`
- `entrypoints-integration-responses-api`
- `entrypoints-integration-speech_to_text`
- `entrypoints-unit-tests`
- `examples`
- `fusion-e2e-config-sweep-h100`
- `fusion-e2e-quick-h100`
- `fusion-e2e-tp2-ar-rms-config-sweep-h100`
- `fusion-e2e-tp2-ar-rms-dynamo-partition-amd`
- `fusion-e2e-tp2-ar-rms-inductor-partition-amd`
- `fusion-e2e-tp2-b200`
- `fusion-e2e-tp2-quick-h100`
- `hybrid-ssm-nixlconnector-pd-accuracy-tests-4-gpus`
- `inkling-unit-tests-b200`
- `kernels-attention-test` ×7
- `kernels-b200` ×3
- `kernels-core-operation-test` ×3
- `kernels-mhc-test-b200`
- `kernels-root-misc-test-b200`
- `kimi-k3-prefix-cache-4xb200` ×2
- `kimi-k3-unit-tests-b200`
- `language-models-tests-extra-standard` ×2
- `language-models-tests-granite-l4-compatibility`
- `language-models-tests-hybrid` ×2
- `language-models-tests-standard`
- `lm-eval-dspark-watermark-2xh100`
- `lm-eval-turboquant-k3v4nc`
- `lm-eval-turboquant-k8v4`
- `lm-eval-turboquant-t3nc`
- `lm-eval-turboquant-t4nc`
- `lm-eval-watermarking`
- `lora` ×4
- `lora-tp-distributed` ×4
- `metrics-tracing-2-gpus`
- `model-executor`
- `mooncake-ec-tcp-e2e-2-gpus`
- `mrcr-eval-small-models`
- `multi-modal-models-standard-1-qwen2`
- `multi-modal-models-standard-2-qwen3-gemma`
- `multi-modal-models-standard-3-llava-qwen2-vl`
- `multi-modal-models-standard-4-other-whisper`
- `multi-modal-processor` ×4
- `multi-modal-processor-cpu` ×4
- `openai-api-correctness`
- `pipeline-context-parallelism-4-gpus`
- `plugin-tests-2-gpus`
- `pytorch-compilation-dynamic-shapes`
- `pytorch-compilation-unit-tests`
- `pytorch-compilation-unit-tests-h100`
- `pytorch-fullgraph-cudagraph-l4-compatibility`
- `pytorch-fullgraph-test`
- `quantization` ×4
- `quantized-moe-test-b200`
- `qwen4-exp-unit-tests-cpu`
- `rayexecutorv2-4-gpus`
- `regression`
- `replayssm-e2e`
- `rust-frontend-core-correctness`
- `rust-frontend-serve-admin-coverage`
- `samplers-multimodal-beam-search`
- `samplers-test`
- `spec-decode-draft-model` ×4
- `spec-decode-eagle-1-deepseek-qwen`
- `spec-decode-eagle-2-llama3-qwen-vl-other`
- `spec-decode-mtp-deepseek-mimo`
- `spec-decode-mtp-gemma4`
- `spec-decode-mtp-qwen3-5`
- `spec-decode-speculators`
- `v1-attention-b200` ×2
- `v1-attention-h100-mi300` ×2
- `v1-core`
- `v1-executor-worker`
- `v1-kv-connectors` ×4
- `v1-metrics-lmeval`
- `v1-others-cpu`
- `v1-sample`
- `v1-spec-decode`

</details>

<details><summary>Would skip (today's rules run them) (19)</summary>

- `ascend-npu-test`
- `basic-correctness-cumem`
- `basic-models-test-other-cpu`
- `cpu-tool-parsers`
- `distributed-compile-rpc-tests-2-gpus`
- `entrypoints-integration-api-server-generate`
- `entrypoints-integration-api-server-openai-chat_completion`
- `entrypoints-integration-api-server-openai-completion`
- `entrypoints-integration-multimodal`
- `fault-tolerance-e2e-2xh100`
- `kernels-fla-ops-test-b200`
- `kernels-fusedmoe-layer-test-2-b200s`
- `kernels-fusedmoe-layer-test-2-h100s`
- `kernels-moe-test` ×5
- `pytorch-compilation-passes-unit-tests`
- `qwen4-exp-unit-tests`
- `rust-frontend-distributed`
- `v1-kv-offload`
- `v1-logits-oracle`

</details>

<details><summary>Would add (today's rules do not run them) (56)</summary>

- `amd-kernels-mi355` (Python record)
- `arm-cpu-test` ×3 (code map)
- `basic-models-tests-extra-initialization` ×14 (code map)
- `batch-invariance-b200` (Python record)
- `batch-invariance-h100` (Python record)
- `cpu-kernel-tests` ×2 (code map)
- `cpu-multi-modal-model-tests-n` ×4 (code map)
- `deepseek-v4-kernel-test-b200` (code map)
- `deepseek-v4-kernel-test-h100` (code map)
- `distributed-tests-8xh100` (Python record)
- `distributed-torchrun-examples-4-gpus` (Python record)
- `elastic-ep-scaling-test` (Python record)
- `engine-1-gpu` (Python record)
- `entrypoints-unit-tests` (Python record)
- `examples` (Python record)
- `fusion-e2e-config-sweep-h100` (Python record)
- `fusion-e2e-quick-h100` (Python record)
- `fusion-e2e-tp2-ar-rms-config-sweep-h100` (Python record)
- `fusion-e2e-tp2-ar-rms-dynamo-partition-amd` (Python record)
- `fusion-e2e-tp2-ar-rms-inductor-partition-amd` (Python record)
- `fusion-e2e-tp2-b200` (Python record)
- `fusion-e2e-tp2-quick-h100` (Python record)
- `hybrid-ssm-nixlconnector-pd-accuracy-tests-4-gpus` (Python record)
- `inkling-unit-tests-b200` (code map)
- `kernels-attention-test` ×7 (code map)
- `kernels-b200` ×3 (Python record)
- `kernels-core-operation-test` ×3 (code map)
- `kimi-k3-prefix-cache-4xb200` ×2 (Python record)
- `kimi-k3-unit-tests-b200` (code map)
- `language-models-tests-extra-standard` ×2 (code map)
- `lm-eval-turboquant-k3v4nc` (Python record)
- `lm-eval-turboquant-k8v4` (Python record)
- `lm-eval-turboquant-t3nc` (Python record)
- `lm-eval-turboquant-t4nc` (Python record)
- `lora` ×4 (Python record)
- `lora-tp-distributed` ×4 (Python record)
- `model-executor` (code map)
- `mrcr-eval-small-models` (Python record)
- `openai-api-correctness` (Python record)
- `plugin-tests-2-gpus` (Python record)
- `quantization` ×4 (code map)
- `quantized-moe-test-b200` (Python record)
- `rayexecutorv2-4-gpus` (code map)
- `rust-frontend-core-correctness` (Python record)
- `rust-frontend-serve-admin-coverage` (Python record)
- `samplers-multimodal-beam-search` (Python record)
- `samplers-test` (Python record)
- `spec-decode-draft-model` ×4 (Python record)
- `spec-decode-eagle-1-deepseek-qwen` (Python record)
- `spec-decode-eagle-2-llama3-qwen-vl-other` (Python record)
- `spec-decode-mtp-deepseek-mimo` (Python record)
- `spec-decode-mtp-gemma4` (Python record)
- `spec-decode-mtp-qwen3-5` (Python record)
- `spec-decode-speculators` (Python record)
- `v1-attention-b200` ×2 (code map)
- `v1-attention-h100-mi300` ×2 (code map)

</details>

<details><summary>AMD mirrors: would skip (17)</summary>

- `basic-correctness-cumem`
- `deepseek-v4-kernel-test-b200`
- `distributed-compile-rpc-tests-2-gpus`
- `entrypoints-integration-api-server-generate`
- `entrypoints-integration-api-server-openai-chat_completion`
- `entrypoints-integration-api-server-openai-completion`
- `entrypoints-integration-multimodal`
- `fault-tolerance-e2e-2xh100`
- `kernels-fla-ops-test-b200`
- `kernels-fp4-moe-test-b200`
- `kernels-fusedmoe-layer-test-2-b200s`
- `kernels-fusedmoe-layer-test-2-h100s`
- `platform-tests`
- `pytorch-compilation-passes-unit-tests`
- `rust-frontend-distributed`
- `v1-kv-offload`
- `v1-logits-oracle`

</details>

<details><summary>AMD mirrors: would add (42)</summary>

- `basic-models-tests-extra-initialization` ×14 (code map)
- `distributed-tests-8xh100` (Python record)
- `distributed-torchrun-examples-4-gpus` (Python record)
- `engine-1-gpu` (code map)
- `entrypoints-unit-tests` (Python record)
- `examples` (Python record)
- `fusion-e2e-config-sweep-h100` (Python record)
- `fusion-e2e-quick-h100` (Python record)
- `fusion-e2e-tp2-ar-rms-config-sweep-h100` (Python record)
- `fusion-e2e-tp2-asynctp-config-sweep-h100` (code map)
- `fusion-e2e-tp2-b200` (Python record)
- `fusion-e2e-tp2-quick-h100` (Python record)
- `hybrid-ssm-nixlconnector-pd-accuracy-tests-4-gpus` (Python record)
- `kernels-attention-test` ×7 (code map)
- `kernels-core-operation-test` ×3 (code map)
- `kimi-k3-unit-tests-b200` (code map)
- `language-models-tests-extra-standard` ×2 (code map)
- `lm-eval-turboquant-k3v4nc` (Python record)
- `lm-eval-turboquant-k8v4` (Python record)
- `lm-eval-turboquant-t3nc` (Python record)
- `lm-eval-turboquant-t4nc` (Python record)
- `lora` ×4 (Python record)
- `lora-tp-distributed` ×4 (Python record)
- `model-executor` (code map)
- `mrcr-eval-small-models` (Python record)
- `openai-api-correctness` (Python record)
- `plugin-tests-2-gpus` (Python record)
- `quantization` ×4 (code map)
- `quantized-models-test` (Python record)
- `rayexecutorv2-4-gpus` (code map)
- `rust-frontend-core-correctness` (Python record)
- `rust-frontend-serve-admin-coverage` (Python record)
- `samplers-test` (Python record)
- `spec-decode-draft-model` ×4 (Python record)
- `spec-decode-eagle-1-deepseek-qwen` (Python record)
- `spec-decode-eagle-2-llama3-qwen-vl-other` (Python record)
- `spec-decode-mtp-deepseek-mimo` (Python record)
- `spec-decode-mtp-gemma4` (Python record)
- `spec-decode-mtp-qwen3-5` (Python record)
- `spec-decode-speculators` (Python record)
- `v1-attention-b200` ×2 (code map)
- `v1-attention-h100-mi300` ×2 (code map)

</details>

<sub>10 changed files · base `b2863c0de9` · head `0a9ae8147a` · Python record: build 92838 at `92044241a0` · kernel record: table [`9204424`](https://github.com/aarnphm/aarnphm.github.io/commit/92044241a0) (build 92838), map [`9204424`](https://github.com/aarnphm/aarnphm.github.io/commit/92044241a0) · not counted: 10 build steps, 5 A100 steps the generator no longer emits, 72 optional steps the selector would also run</sub>

**Juntian777** (2026-10-06T00:22:38Z):

/ci run

**github-actions** (2026-10-06T00:24:02Z):

✅ Triggered [Buildkite CI #92990](https://buildkite.com/vllm/ci/builds/92990) for commit `0a9ae8147a5b`.

**Juntian777** (2026-10-06T01:56:18Z):

/ci retry

**github-actions** (2026-10-06T01:56:36Z):

✅ Queued 1 failed job(s) for retry in [Buildkite CI #92990](https://buildkite.com/vllm/ci/builds/92990).

