---
title: '57097'
pageLayout: default
slug: thoughts/inference/vllm-prs/57097
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/57097.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#57097](https://github.com/aarnphm/aarnphm.github.io/issues/57097) \[Qwen3.8-Flash-Next\] Fuse main QK-norm/RoPE/gate and KV-cache write into the QSA pre-indexer launch

[vllm-project/vllm#57097](https://github.com/vllm-project/vllm/pull/57097) · ShuoleiWang · MERGED · created 2026-09-16T01:56:18Z · merged 2026-10-01T06:21:07Z

## Purpose

Each QSA layer of Qwen3.8-Flash-Next prepares its inputs with three launches before the sparse attention kernel:

1. `fused_qk_rmsnorm_rope_gate`: main-attention Q/K RMSNorm, RoPE and the gate split.
2. `qsa_pre_indexer`: indexer Q/K norm and RoPE, compressed-K cache and raw-key ring.
3. `reshape_and_cache_flash`: main K/V cache write.

This PR appends the main-attention work to the `qsa_pre_indexer` launch and renames it to `qsa_prepare`. After the indexer programs, one program per (token, Q or KV head) normalizes and rotates Q/K, copies the gate and writes K/V straight into the paged cache: BF16, or FP8 scaled by the layer’s `k_scale`/`v_scale` the way `reshape_and_cache_flash` does. Two launches and the round trip of K through an intermediate tensor go away. The fused path runs for every batch (decode, prefill, mixed) whenever both fused kernels are available; otherwise the existing path runs.

`vllm/model_executor/layers/fused_qk_norm_rope.py` is unchanged; the logic lives in `vllm/models/qwen4_exp/nvidia/ops/qsa_prepare.py` (renamed from `qsa_pre_indexer.py`).

## Test Plan



```bash
pytest tests/models/qwen4_exp/test_qsa_pre_indexer.py
pytest tests/models/qwen4_exp/test_qsa_reference.py tests/models/qwen4_exp/test_config.py
```

`test_qsa_fused_pre_indexer_matches_unfused` now also checks the main Q, gate and K/V cache against `fused_qk_rmsnorm_rope_gate` + `reshape_and_cache_flash` over the existing decode, mixed and 4097-token prefill layouts, with BF16 and FP8 caches.

End to end, the fused path is the default, so the existing accuracy eval `tests/evals/qwen4_exp/test_accuracy.py` (the optional H200 and B200 steps in `.buildkite/test_areas/lm_eval.yaml`) runs through it without changes.

Microbenchmark per `.agents/skills/kernel-microbenchmark`: FlashInfer CUPTI timing with CUDA graph and cold L2, outputs checked against the three kernels before timing, arms alternated. Serving: GSM8K and `vllm bench serve` with the recipe’s FP8 configuration, main vs. this PR.

## Test Result

8x H20, main `741edeebee`, torch 2.13.0+cu130, Triton 3.7.1. The branch is now rebased on `72e7874fa6`; none of the 82 commits in between touch the changed files or the QSA code (the only change under `vllm/models/qwen4_exp/` is the PLE prefetch buffer, [#58797](https://github.com/aarnphm/aarnphm.github.io/issues/58797)). The microbenchmark and serving tables were measured on the previous revision of this PR; the last paragraph of this section covers the re-run after the latest review changes.

**Tests:** 16 passed (`test_qsa_pre_indexer.py`); 82 passed (`test_qsa_reference.py`, `test_config.py`).

**Kernel microbenchmark, one H20 GPU.** What is timed is the prepare stage of one QSA layer, from the projection outputs until Q, the gate and all caches are written. On main that is three kernels (`fused_qk_rmsnorm_rope_gate`, `qsa_pre_indexer`, `reshape_and_cache_flash`); with this PR it is one `qsa_prepare` launch.

- M is the number of tokens in the batch. Decode batches are requests × 4 tokens (1 + 3 MTP draft tokens), each request with an 8K-token context; prefill batches are one request with M prompt tokens.
- Heads per GPU are those of TP8: the model has 24 query heads and 2 KV heads, so each GPU has 3 query heads and 1 KV head.
- The main KV cache and the indexer cache are FP8, as in the recipe; BF16 caches are in the details below.
- Time saved = 1 − fused time / three-kernel time. The three kernels are timed at their fastest `num_warps`: `fused_qk_rmsnorm_rope_gate` ships with 4 warps for head_dim 256, and 1 warp was faster at every M. Against the shipped setting the fused kernel saves 56-70%. The fused kernel keeps the pre-indexer’s single warp, which beat 2 and 4 warps at M=16 and M=16384.

| M (tokens) | Batch                           | Three kernels (µs) | Fused (µs) | Time saved |
| ---------: | ------------------------------- | -----------------: | ---------: | ---------: |
|          1 | decode, 1 request × 1 token     |               10.4 |        4.2 |      60.0% |
|          4 | decode, 1 request × 4 tokens    |               10.7 |        4.2 |      61.0% |
|         16 | decode, 4 requests × 4 tokens   |               10.8 |        4.1 |      62.0% |
|         64 | decode, 16 requests × 4 tokens  |               11.1 |        4.3 |      61.4% |
|        256 | decode, 64 requests × 4 tokens  |               12.1 |        4.8 |      60.4% |
|       1024 | decode, 256 requests × 4 tokens |               21.8 |        7.7 |      64.5% |
|       1024 | prefill, 1 request              |               23.7 |        8.1 |      65.7% |
|       4096 | prefill, 1 request              |               57.0 |       22.6 |      60.3% |
|      16384 | prefill, 1 request              |              183.2 |       84.7 |      53.7% |

With BF16 caches the fused kernel saves 57-59% for decode batches and 37% for the 16384-token prefill.

<details>
<summary>Full numbers, FP8 and BF16 caches</summary>

“As shipped” is the three kernels with `fused_qk_rmsnorm_rope_gate` at its default 4 warps. GB/s is the minimum number of bytes the operation must move (every input read once, every output written once) divided by the time; the three kernels’ GB/s uses their fastest-warps time. H20’s HBM peak is about 4 TB/s.

#### FP8 caches

| M (tokens) | Batch        | Three kernels, as shipped (µs) | Three kernels, fastest warps (µs) | Fused (µs) | Time saved vs fastest | Fused GB/s | Three kernels GB/s |
| ---------: | ------------ | -----------------------------: | --------------------------------: | ---------: | --------------------: | ---------: | -----------------: |
|          1 | decode 1×1   |                           10.7 |                              10.4 |        4.2 |                 60.0% |          2 |                  1 |
|          4 | decode 1×4   |                           11.0 |                              10.7 |        4.2 |                 61.0% |          9 |                  4 |
|         16 | decode 4×4   |                           10.9 |                              10.8 |        4.1 |                 62.0% |         38 |                 15 |
|         64 | decode 16×4  |                           11.6 |                              11.1 |        4.3 |                 61.4% |        146 |                 56 |
|        256 | decode 64×4  |                           12.5 |                              12.1 |        4.8 |                 60.4% |        523 |                207 |
|       1024 | decode 256×4 |                           24.7 |                              21.8 |        7.7 |                 64.5% |       1297 |                460 |
|       1024 | prefill      |                           26.6 |                              23.7 |        8.1 |                 65.7% |       1199 |                412 |
|       4096 | prefill      |                           71.2 |                              57.0 |       22.6 |                 60.3% |       1726 |                685 |
|      16384 | prefill      |                          247.8 |                             183.2 |       84.7 |                 53.7% |       1842 |                852 |

#### BF16 caches

| M (tokens) | Batch        | Three kernels, as shipped (µs) | Three kernels, fastest warps (µs) | Fused (µs) | Time saved vs fastest | Fused GB/s | Three kernels GB/s |
| ---------: | ------------ | -----------------------------: | --------------------------------: | ---------: | --------------------: | ---------: | -----------------: |
|          1 | decode 1×1   |                            9.9 |                               9.5 |        4.0 |                 58.2% |          3 |                  1 |
|          4 | decode 1×4   |                           10.3 |                              10.1 |        4.2 |                 58.3% |         10 |                  4 |
|         16 | decode 4×4   |                           10.2 |                               9.9 |        4.2 |                 57.4% |         41 |                 18 |
|         64 | decode 16×4  |                           10.8 |                              10.5 |        4.4 |                 58.4% |        159 |                 66 |
|        256 | decode 64×4  |                           11.7 |                              11.3 |        4.9 |                 57.1% |        572 |                246 |
|       1024 | decode 256×4 |                           21.0 |                              17.9 |        7.4 |                 58.9% |       1512 |                621 |
|       1024 | prefill      |                           22.8 |                              19.5 |        8.0 |                 58.9% |       1353 |                556 |
|       4096 | prefill      |                           54.6 |                              40.4 |       21.2 |                 47.5% |       2040 |               1072 |
|      16384 | prefill      |                          188.8 |                             130.6 |       82.8 |                 36.6% |       2095 |               1328 |

</details>

**Serving** (8x H20, TP8 + EP8, MTP3, Triton MoE):

| Cache dtype  |   GSM8K, main | GSM8K, this PR | Identical predictions |
| ------------ | ------------: | -------------: | --------------------: |
| FP8 (recipe) | 96.97% (1279) |  96.97% (1279) |             1301/1319 |
| BF16         | 96.97% (1279) |  96.66% (1275) |             1304/1319 |

GSM8K is 5-shot, 1319 questions, greedy, 64 concurrent, one run per arm; the cache dtype applies to both the main KV cache and the indexer cache. For scale, stock FP8 and stock BF16 differ from each other on 17 questions.

Throughput, `vllm bench serve` random 8192/1024 with FP8 caches, one run each: C1 260.7 → 266.8 output tok/s (+2.3%), C64 1636.3 → 1759.4 (+7.5%). These are single-run sanity checks and do not establish an end-to-end throughput or accuracy difference in either direction. For reference, the kernel savings add up to about 1% of the run time at C1 and 0.3% at C64.

The latest review changes move arguments and buffer allocation around and keep the kernel math. On the final code, the QSA tests pass (16 + 82) and the fused kernel time is unchanged (4.18 µs at M=16, 83.2 µs at M=16384). A single GSM8K run with FP8 caches gave 96.59% (1274) vs. 96.97% (1279) on main, with 1303/1319 predictions identical; earlier single runs of the same kernel math on previous revisions gave 1279 and 1281.

Only H20 was measured. The change removes two launches and one write and read of K per token, and adds no other work.

AI assistance: developed with Claude Code; I reviewed every changed line and ran the tests and benchmarks above.

---

## comments

**ShuoleiWang** (2026-09-16T11:54:22Z):

Close this draft PR since reviewer think the performance gain is too low

**ShuoleiWang** (2026-09-30T01:53:13Z):

Reopen this PR and here is the update:

- Rebased on latest main, with the changes as one new commit on top of the original one. [#55557](https://github.com/aarnphm/aarnphm.github.io/issues/55557) landed in the meantime and the recipe now uses an FP8 main KV cache, so the fused path also writes FP8 K/V, with the same per-tensor scaling as reshape_and_cache_flash.
- Removed the num_prefills == 0 guard; decode, prefill and mixed batches all use the fused launch.
- fused_qk_norm_rope.py is untouched. The main-attention logic is inlined in qsa_pre_indexer.py; head_dim must be a power of 2, and the block sizes and rotary width are derived in the kernel.
- The test only extends test_qsa_fused_pre_indexer_matches_unfused with a fuse_main correctness check (BF16 and FP8 caches); no CUDA graph test.
- Microbenchmark per the skill (CUPTI, CUDA graph, cold L2), M=1 to 16384 on H20: the fused launch takes 54-66% less time than the three kernels with FP8 caches and 37-59% with BF16 caches, with the three kernels at their fastest warp count. Tables are in the description.

**gau-nernst** (2026-10-01T03:50:53Z):

Independent e2e A/B on 4x GB300 (the PR’s measurements are H20; kernel unchanged since then). Baseline = the PR’s ancestor commit on main (`17e9295dd5`), PR = `1a172e3b78`; same node, sequential arms, identical serving flags.

Setup: `Qwen/Qwen3.8-Flash-Next-FP8`, TP4 + EP4, MTP3, `--kv-cache-dtype fp8`, fp8 indexer cache, no prefix caching. `vllm bench serve` SPEED-Bench 8192/1024, seed 0, concurrency sweep, single run per arm.

| Concurrency | Output tok/s, main → PR (Δ) | TTFT p50 (Δ) | TPOT p50 (Δ) | TPOT p99 (Δ) | E2EL p50 (Δ) | MTP accept len |
| ----------: | --------------------------- | ------------ | ------------ | ------------ | ------------ | -------------- |
|           1 | 293 → 298 (+1.5%)           | −5.8%        | −1.5%        | +0.4%        | −1.7%        | 2.11 → 2.10    |
|           4 | 822 → 846 (+2.9%)           | −4.4%        | −1.4%        | −5.4%        | −2.4%        | 2.06 → 2.04    |
|          16 | 1934 → 1972 (+1.9%)         | −3.6%        | −1.7%        | −2.3%        | −1.4%        | 2.06 → 2.03    |
|          64 | 3605 → 3679 (+2.0%)         | −1.2%        | −2.4%        | −0.3%        | −2.5%        | 2.04 → 2.05    |

Takeaway: PR is faster at every concurrency level and MTP acceptance is unchanged, i.e. no e2e regression and a likely small win. Single run per arm, so treat exact magnitudes as noise-level; the direction is consistent with the kernel microbenchmarks (fused prepare ≈ 60% less time than the three-kernel path at decode shapes, reproduced on GB300). The QSA unit tests (16 + 82) also pass on GB300.

(Independent validation by @gau-nernst’s agent; no affiliation with the PR author.)

**gau-nernst** (2026-10-01T04:47:21Z):

/ci run

**github-actions** (2026-10-01T04:47:23Z):

✅ @ShuoleiWang, CI is now available for this PR.

- `/ci run` starts upstream CI; `/amd-ci run` starts AMD CI only.
- Your branch must contain every commit currently on its upstream target branch. Merge or rebase onto the latest target branch, then rerun the command. Append `--allow-stale` to a run command to test an outdated branch at your own risk.
- `/ci retry` retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
- `/amd-ci retry` retries failed jobs in AMD CI for the current PR head. Use `/amd-ci run` when the current head has no AMD CI build.
- `/ci cancel` cancels scheduled or running CI builds for this PR branch; `/amd-ci cancel` does the same for AMD CI only.

<!-- vllm-ci-authorized -->

**github-actions** (2026-10-01T04:47:41Z):

✅ Triggered [Buildkite CI #92342](https://buildkite.com/vllm/ci/builds/92342) for commit `de2a5c488545`.

