---
title: '57407'
pageLayout: default
slug: thoughts/inference/vllm-prs/57407
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/57407.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#57407](https://github.com/aarnphm/aarnphm.github.io/issues/57407) \[ROCm\]\[Perf\] Enable layer-aware CSA2 multi-stream overlap for DeepSeek-V4.1-Flash

[vllm-project/vllm#57407](https://github.com/vllm-project/vllm/pull/57407) · shen-shanshan · MERGED · created 2026-09-17T15:42:14Z · merged 2026-09-27T15:48:02Z

## Purpose

DeepSeek-V4.1-Flash has three attention layer shapes: full CSA2 layers (2/8/14/20) run SWA + compressor + indexer and build a new global KV cache with their own Top-K indices; reindex CSA2 layers (24/28/32/36) run SWA + indexer only, re-scoring Top-K over an earlier layer’s global KV cache; all other layers are dense or pure SWA.

The shared v4.1 pipeline overlaps the full CSA2 layers with **three** fork/join pairs per step (input projections; Q/SWA-insert vs compressor; indexer vs cache-insert) and the reindex layers with one. On ROCm each fork/join is a HIP event pair per side stream, so the multi-stream launch cost outweighs the overlap benefit there.

---

This PR gives the ROCm attention layer a layer-type-aware scheme with exactly one fork/join per step:

**Full CSA2 layers (`_forward_csa2_full`)**: one 3-stream fork right after the input norm.

- default stream: fused wqa+wkv → q/kv split+norm → wq_b → fused qnorm/RoPE/SWA-insert;
- aux stream 0: the compressor chain (kv_score GEMM → save/compress/norm → RoPE-quant cache insert);
- aux stream 1: indexer `weights_proj`.

After the single join, the indexer K write (`_produce_k`, whose `wk` consumes the aux-0 latent) and the query path (`forward_q`, whose wq_b consumes the default-stream qr) run serially, followed by the indexer op and sparse MLA.

**Reindex CSA2 layers (`_forward_csa2_reindex`)**: the input projections run serially first (the query path consumes qr, so overlapping the weight projection is not worth it), then one 2-stream fork overlaps the SWA q path with the indexer query path.

**Dense layers**: fully serial (`aux_stream_list` disabled).

---

A new shared helper `execute_in_parallel_default_first` (in `vllm/utils/multi_stream_utils.py`) mirrors `execute_in_parallel` but enqueues the default chain before the aux chains — the default chain is the longest, so starting it first shortens the step tail while the aux streams still only wait on the pre-fork default-stream work.

The fork is gated exactly like the shipped v4.0 multi-stream gate ([#51794](https://github.com/aarnphm/aarnphm.github.io/issues/51794)/[#56853](https://github.com/aarnphm/aarnphm.github.io/issues/56853)): it only runs where the fork/join becomes static graph edges (inside capture, or with non-dict metadata such as the profile run); everywhere else — including MRV1, which keeps its wide eager region ([#51430](https://github.com/aarnphm/aarnphm.github.io/issues/51430)) — the layer falls back to the fully sequential base pipeline.

The CUDA/NVIDIA path is untouched except for a behavior-neutral base refactor: the v4.1 indexer’s query path is extracted into `DeepseekV4Indexer.forward_q`, still driven unchanged by the base `forward` (`_produce_k` keeps its private name). This is not a duplicate of [#51794](https://github.com/aarnphm/aarnphm.github.io/issues/51794)/[#56853](https://github.com/aarnphm/aarnphm.github.io/issues/56853), which cover v4.0: v4.1-flash has a different topology (the indexer derives K from the kv-source compressor’s latent, with no indexer-local compressor), so the overlap must differ per layer type.

## Trace Comparison (Take conc=4 as an example)

**With regard to CSA2 layers with compressor and indexer (layer_id=2/8/14/20):**

Before:

<img width="2198" height="381" alt="main-1" src="https://github.com/user-attachments/assets/c9cb4a17-d4e6-4f03-87e7-b40042b1b27e" />

> The default NV path launches multi-stream 3 times per layer, which is inefficient on ROCm path (multi-stream cost > overlap benefit).

After (**20.76%** faster than the old pattern):

<img width="2184" height="380" alt="csa-1" src="https://github.com/user-attachments/assets/fa226455-9069-44f3-8c47-3faf4edb6839" />

---

**With regard to CSA2 layers only with indexer (layer_id=24/28/32/36):**

Before:

<img width="2193" height="383" alt="main-2" src="https://github.com/user-attachments/assets/f9f71402-b810-4ecc-9d90-a8bb1224d64a" />

> No need to overlap indexer weight projection, since the benefit (4 μs) < the multi-stream cost (5 μs + 6 μs = 11 μs). Instead, we prefer to overlap SWA process and indexer process.

After (**11.90%** faster than the old pattern):

<img width="2192" height="416" alt="csa-2" src="https://github.com/user-attachments/assets/6d2908ee-d7d1-4abd-aea4-4f9d2ac34eb9" />

## Test Plan

Benchmark with SA InferenceX `8k1k` workload with server launch commands below:



```bash
export VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_MOE=1
export AITER_TRITON_LOG_LEVEL=ERROR VLLM_USE_BREAKABLE_CUDAGRAPH=1
export OMP_NUM_THREADS=1 VLLM_ENGINE_READY_TIMEOUT_S=3600
export VLLM_USE_RUST_FRONTEND=1 PYTHONUNBUFFERED=1 GPU_COUNT=4
export HF_HOME=/shared/models

vllm serve deepseek-ai/DeepSeek-V4.1-Flash \
    --host 0.0.0.0 \
    --port 8333 \
    --tensor-parallel-size 4 \
    --language-model-only \
    --tokenizer-mode deepseek_v41 \
    --tool-call-parser deepseek_v41 \
    --enable-auto-tool-choice \
    --reasoning-parser deepseek_v41 \
    --moe-backend aiter \
    --gpu-memory-utilization 0.9 \
    --max-model-len 1048576 \
    --max-num-seqs 128 \
    --max-cudagraph-capture-size 1024 \
    --max-num-batched-tokens 16384 \
    --disable-uvicorn-access-log \
    --no-swa-bounded-replay \
    --engram-config '{"cpu_offload":false}' \
    2>&1 | tee "${OUTDIR}/server.log"
```

Accuracy test with `gsm8k` dataset using `lm_eval`:



```bash
lm_eval \
  --model local-completions \
  --model_args model=deepseek-ai/DeepSeek-V4.1-Flash,base_url=http://0.0.0.0:8333/v1/completions,num_concurrent=64,max_retries=10,max_gen_toks=2048,timeout=60000,tokenized_requests=False \
  --batch_size auto \
  --tasks gsm8k \
  --num_fewshot 5 \
  --output_path . 2>&1 | tee -a eval.log
```

> \[!note\]
>
> All tests based on latest `nightly-rocm100` image.

## Test Result

### Benchmark

By the way, note that the e2e benefit could be small, since there are onlt **8** layers that we can optimize (overlap), while other layers always execute serially.

| Conc | Version | Output token throughput (tok/s) |  Change | Mean TTFT (ms) |  Change  | Mean TPOT (ms) |  Change |
| ---: | :------ | ------------------------------: | :-----: | -------------: | :------: | -------------: | :-----: |
|    1 | main    |                          114.47 |    —    |         392.61 |     —    |           8.32 |    —    |
|    1 | this PR |                          117.28 | 2.45% ↑ |         266.06 | 32.23% ↓ |           8.25 | 0.84% ↓ |
|    4 | main    |                          381.72 |    —    |         348.33 |     —    |           9.86 |    —    |
|    4 | this PR |                          386.47 | 1.24% ↑ |         331.76 |  4.76% ↓ |           9.76 | 1.01% ↓ |
|   16 | main    |                         1018.13 |    —    |         496.96 |     —    |          14.73 |    —    |
|   16 | this PR |                         1018.69 | 0.06% ↑ |         495.02 |  0.39% ↓ |          14.73 |  0.00%  |
|   64 | main    |                         1912.44 |    —    |        1165.33 |     —    |          31.71 |    —    |
|   64 | this PR |                         1920.10 | 0.40% ↑ |        1160.00 |  0.46% ↓ |          31.59 | 0.38% ↓ |

In summary, the performance data indicates that this PR provides consistent throughput gains and latency reductions without introducing a regression in the tested metrics. The most noticeable benefit is at low concurrency, while at higher concurrency (conc=16 and 64) the changes are comparatively small, suggesting that the optimization has limited additional impact once the system becomes heavily loaded.

### Accuracy Test

Main:

| Tasks | Version | Filter           | n-shot | Metric      |   |  Value |   | Stderr |
| ----- | ------: | ---------------- | -----: | ----------- | - | -----: | - | -----: |
| gsm8k |       3 | flexible-extract |      5 | exact_match | ↑ | 0.9363 | ± | 0.0067 |
|       |         | strict-match     |      5 | exact_match | ↑ | 0.9363 | ± | 0.0067 |

This PR:

| Tasks | Version | Filter           | n-shot | Metric      |   |  Value |   | Stderr |
| ----- | ------: | ---------------- | -----: | ----------- | - | -----: | - | -----: |
| gsm8k |       3 | flexible-extract |      5 | exact_match | ↑ | 0.9363 | ± | 0.0067 |
|       |         | strict-match     |      5 | exact_match | ↑ | 0.9371 | ± | 0.0067 |

---

<details>
<summary> Essential Elements of an Effective PR Description Checklist </summary>

- [x] The purpose of the PR, such as “Fix some issue (link existing issues this PR will resolve)”.
- [x] The test plan, such as providing test command.
- [x] The test results, such as pasting the results comparison before and after, or e2e results
- [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.

</details>

---

## comments

**shen-shanshan** (2026-09-24T11:28:30Z):

/ci run

**github-actions** (2026-09-24T11:28:48Z):

❌ This PR is 1 commit behind upstream `main`. Your branch must contain every commit currently on upstream `main`. No new CI build was started. Merge or rebase onto the latest `main`, then rerun `/ci run`. To test this branch at your own risk, use `/ci run --allow-stale`.

<!-- vllm-ci-command:5813169885 -->

**shen-shanshan** (2026-09-24T11:29:55Z):

/ci run

**github-actions** (2026-09-24T11:30:16Z):

✅ Triggered [Buildkite CI #90931](https://buildkite.com/vllm/ci/builds/90931) for commit `6bb35834177d`.

**shen-shanshan** (2026-09-24T16:13:03Z):

/ci retry

**github-actions** (2026-09-24T16:13:25Z):

✅ Queued 1 failed job(s) for retry in [Buildkite CI #90931](https://buildkite.com/vllm/ci/builds/90931).

**Fangzhou-Ai** (2026-09-25T01:21:25Z):

LGTM, qq do we need an UT here? It’s non-blocking so let’s run CI now.

**shen-shanshan** (2026-09-25T08:14:40Z):

> LGTM, qq do we need an UT here? It’s non-blocking so let’s run CI now.

Test added.

**shen-shanshan** (2026-09-25T08:15:08Z):

/ci run

**github-actions** (2026-09-25T08:15:27Z):

❌ This PR is 1 commit behind upstream `main`. Your branch must contain every commit currently on upstream `main`. No new CI build was started. Merge or rebase onto the latest `main`, then rerun `/ci run`. To test this branch at your own risk, use `/ci run --allow-stale`.

<!-- vllm-ci-command:5829188261 -->

**shen-shanshan** (2026-09-25T08:16:01Z):

/ci run

**github-actions** (2026-09-25T08:16:20Z):

✅ Triggered [Buildkite CI #91183](https://buildkite.com/vllm/ci/builds/91183) for commit `da41c629fef3`.

**shen-shanshan** (2026-09-25T08:18:44Z):

CC @tjtanaa

**shen-shanshan** (2026-09-25T13:57:39Z):

/ci run

**github-actions** (2026-09-25T13:57:59Z):

✅ Triggered [Buildkite CI #91206](https://buildkite.com/vllm/ci/builds/91206) for commit `a76638cea342`.

