---
title: '58586'
pageLayout: default
slug: thoughts/inference/vllm-prs/58586
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/58586.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#58586](https://github.com/aarnphm/aarnphm.github.io/issues/58586) \[Kernel\]\[DSV4.1\] Fuse MoE finalize into the TP all-reduce + mHC boundary

[vllm-project/vllm#58586](https://github.com/vllm-project/vllm/pull/58586) · zyongye · MERGED · created 2026-09-24T16:22:16Z · merged 2026-09-26T19:08:37Z

## Purpose

DSV4.1 decode (TP4, `0 < M <= 16`, FULL CUDA graphs) closes every sublayer with a fused TP all-reduce + mHC kernel. After an MoE, that boundary is preceded by FlashInfer’s MoE finalize (top-k weighted sum) and a separate shared-expert add. This PR folds the finalize and the shared add into the boundary:

- **Vendored FlashInfer LL kernels.** `vllm/models/deepseek_v41/nvidia/ops/cute_dsl/` copies FlashInfer’s CuTe DSL low-latency MNNVL all-reduce (`flashinfer/comm/mnnvl_cutedsl/kernel_ll`, [flashinfer-ai/flashinfer@`139af6f`](https://github.com/flashinfer-ai/flashinfer/commit/139af6f6), the H5120 presets):
  - `_SharedOnlyPublishDeviceKernel` and `_group_leader_block_sum` are byte-identical to upstream.
  - `_QuadFinalizePublishDeviceKernel` is upstream’s plus an `fp32_weights` option. DSV4.1’s router emits FP32 top-k weights with the routed scaling folded in; FlashInfer loads BF16.
  - `_LamportMHCDeviceKernel` is upstream’s `_LamportResidualRMSNormDeviceKernel`. Its residual add and copy are replaced by the mHC post-mix of the hc streams (stored as the new residual) and their pre-mix collapse, which the unchanged RMSNorm then normalizes. This replaces `torch.ops._C_custom_ar.all_reduce_mhc` ([#57643](https://github.com/aarnphm/aarnphm.github.io/issues/57643)) on the overlap path. That CUDA kernel has no other user, so this PR deletes `csrc/libtorch_stable/all_reduce_mhc.cu` and its CMake entry. [#57643](https://github.com/aarnphm/aarnphm.github.io/issues/57643)’s boundary wiring (`reduce_results`, `fuse_mhc_all_reduce`, `supports_mhc_all_reduce`) is kept and reused.
- **MoE left unfinalized, via [#58635](https://github.com/aarnphm/aarnphm.github.io/issues/58635).** For layers whose experts are the modular TRTLLM MXFP4 ones, DSV4.1 requests deferral once at model build with `moe_config.defer_moe_finalize(MHC_OVERLAP_MAX_TOKENS)`. Per call, `should_defer_moe_finalize(M)` picks the path, and the experts return a single `UnfinalizedMoEOutput`. This PR touches no fused-MoE code.
- **DSV4.1 wiring (model file only, no new runner).**
  - A layer defers when its successor opens with the fused boundary: 37/40 layers on DSV4.1-Flash. The last layer and the layers before an engram layer finalize as before.
  - `DeepseekV4MoE.forward_unfinalized` runs the runner’s `_forward_impl` directly, since the custom op returns tensors only. It hands `MoEOutput(routed=UnfinalizedMoEOutput, shared_output=...)` to the next layer.
  - `mhc_shifted_post_pre` calls `AllReduceMHC.finalize`, so top-k reduce, shared add, all-reduce and mHC post/pre run as two PDL-chained kernels.
  - Outside FULL graphs (eager/PIECEWISE), the next mixes run inline instead of on the side stream.
  - `MHC_OVERLAP_MAX_TOKENS` (16), the side-stream design and the M > 16 path are unchanged.

Duplicate check: [#58635](https://github.com/aarnphm/aarnphm.github.io/issues/58635) (merged) supplies the modular-path deferral this builds on. [#52601](https://github.com/aarnphm/aarnphm.github.io/issues/52601) (MoE tail all-reduce fusion for MiniMax-M3, last updated 2026-09-02) targets a different model and kernel. No open PR fuses the DSV4.1 MoE finalize into the mHC boundary.

## Test Plan

Unless noted, everything below is on main [`378504a`](https://github.com/aarnphm/aarnphm.github.io/commit/378504a544) (includes [#58635](https://github.com/aarnphm/aarnphm.github.io/issues/58635)) plus this change, with main’s pins: flashinfer 0.7.0, nvidia-cutlass-dsl 4.7.1, torch 2.13.0+cu130. Hardware is DSV4.1-Flash on TP4 GB300.

- Kernel regression test: `tests/distributed/test_custom_all_reduce.py::test_all_reduce_mhc_matches_unfused_path`, which replaces [#57643](https://github.com/aarnphm/aarnphm.github.io/issues/57643)’s CUDA-kernel test. It checks against the unfused math in torch (FP32 rank sum → BF16, mHC post FMA chain → BF16, collapse → BF16, RMSNorm). Run as `pytest -v -s tests/distributed/test_custom_all_reduce.py` with ray workers; all 51 tests in the file passed on 4× GB300. It covers M in \{1, 2, 3, 4, 5, 6, 8, 12, 16\}, visited in shrinking and growing order, ±0 sentinel inputs, CUDA graph replay with mutated inputs, and the finalize path with padded/permuted rows and an invalid (−1) route.
- `pre-commit run` (ruff, typos, SPDX, forbidden imports, test tethering, mypy 3.10) and `pre-commit run mypy-3.12 --hook-stage manual` on the changed files.
- In-engine numerical check (ad-hoc patch, not in this PR) under FULL CUDA graphs.
- E2E perf, `vllm bench serve`: random 1024 in / 256 out, `--ignore-eos`, concurrency 1/4/8/16.
- Accuracy with mqa, using the eval config of this model’s existing recipes. GPQA-Diamond runs at concurrency 16 so decode batches take the fused path.

Server: `vllm serve deepseek-ai/DeepSeek-V4.1-Flash --tensor-parallel-size 4 --kv-cache-dtype fp8 --block-size 128 --max-model-len 16384 --max-num-seqs 64 --max-num-batched-tokens 8192`. GPQA used `--tokenizer-mode deepseek_v41 --reasoning-parser deepseek_v41 --max-model-len 65536 --max-num-seqs 256 --max-num-batched-tokens 16384` instead. The baseline arm is the same tree with deferral disabled, so both arms share the new all-reduce + mHC kernel and differ only in the MoE finalize.

## Test Result

**Correctness**

- Regression test: PASSED.
  - Post-mixed hc streams: bit-identical to the torch reference.
  - Fused-finalize path: bit-identical to an FP32 FMA-chain reference fed through the same collective.
  - Normalized layer input: within 1 BF16 ULP of the torch reference. The RMSNorm is FlashInfer’s, which sums the squares in another order and uses an approximate rsqrt.
  - Before the CUDA kernel was deleted, an earlier revision of this test compared against it directly: the hc streams were bit-identical and the normalized input was within 1 ULP.
- In-engine, FULL graphs: on each rank, the fused boundary was compared against the plain collective run on a torch finalize of the same buffers, inside the same graphs. Over a GSM8K run, 51,393 fused boundaries ran per rank, of which 1,295 were host calls (eager or capture) and the rest graph replays. Both the post-mixed hc streams and the normalized input matched bit-for-bit: 0 of 1.6e10 elements differed, on every rank.
- Before the rebase, an eager check over 55,500 deferred MoE calls found the deferred GEMM2 rows plus an FP32 top-k sum bit-identical to FlashInfer’s own finalized output.

**E2E decode.** Random 1024/256:

| concurrency | median ITL base → PR (ms) | median TPOT base → PR (ms) | output tok/s base → PR  |
| ----------- | ------------------------- | -------------------------- | ----------------------- |
| 1           | 5.14 → 5.03               | 5.15 → 5.05                | 174.7 → 177.8 (+1.8%)   |
| 4           | 5.71 → 5.56               | 5.71 → 5.58                | 581.7 → 591.7 (+1.7%)   |
| 8           | 6.30 → 6.17               | 6.30 → 6.15                | 1069.7 → 1089.8 (+1.9%) |
| 16          | 7.19 → 7.13               | 7.36 → 7.64                | 1818.6 → 1817.6 (flat)  |

At concurrency 16, TPOT moves with prefill interference, so median ITL is the steadier number. Over every c=16 sweep in this work, base median ITL ranged 7.19–7.39 ms (4 sweeps) and fused-finalize 7.13–7.24 ms (5 sweeps, including kernel iterations before the mirror). The pre-rebase sweep measured c=16 at +3.0% throughput.

**Kernel level** (before the rebase; the kernel file is unchanged). One decoder layer, two boundaries each followed by a sublayer, in µs. Each arm is timed as the slope between CUDA graphs of 8 and 40 calls, max over ranks, median of 50 replays:

| M  | sublayers only | before (CUDA all_reduce_mhc, MoE finalizes itself) | CuTe LL, MoE finalizes itself | this PR (fused finalize) | Δ vs before |
| -- | -------------- | -------------------------------------------------- | ----------------------------- | ------------------------ | ----------- |
| 1  | 38.19          | 48.59                                              | 48.15                         | 45.28                    | −3.31       |
| 4  | 70.91          | 82.50                                              | 81.43                         | 79.22                    | −3.28       |
| 8  | 105.36         | 117.61                                             | 116.34                        | 111.61                   | −6.00       |
| 16 | 171.30         | 186.22                                             | 184.91                        | 180.26                   | −5.96       |

**Accuracy**

| eval                                                                                               | base                                  | this PR                               |
| -------------------------------------------------------------------------------------------------- | ------------------------------------- | ------------------------------------- |
| GSM8K (5-shot completions, greedy, n=1319, concurrency 16), 3 runs each                            | 0.9295 / 0.9287 / 0.9272, mean 0.9285 | 0.9249 / 0.9287 / 0.9265, mean 0.9267 |
| GPQA-Diamond (chat, thinking, temp 1.0, top_p 0.95, 57344 tokens, 4 epochs), exact match / pass\@4 | 0.9040 / 0.9545                       | 0.9066 / 0.9444                       |

- Two GSM8K runs of the same code differ on about 46 of 1319 questions, so a single run has an SD of about 0.35 pt.
- Before the rebase, 13 base runs and 20 runs across fused-finalize variants averaged 0.9275 vs 0.9238. A control whose finalize rounds exactly like the unfused path (bit-identical per-rank contribution) trailed by the same amount. The per-question shift was diffuse (net about 5 of 1319, none consistently broken). So that gap was not the fused arithmetic; my best guess is timing-dependent batching, but it is unexplained. On this rebase it is 0.18 pt over 3 runs each.
- GPQA-Diamond: 0 request errors and no empty answers in either arm; 6 (base) and 1 (PR) of 792 generations hit the token cap. pass\@4 differs by 2 of 198 questions. The pre-rebase pair was 0.904 vs 0.904. This model’s dashboard history for the same config (4-epoch runs) has median 0.909 and IQR 0.906–0.913.

## Notes

- Like upstream, `_QuadFinalizePublishDeviceKernel` reads the routing metadata before `griddepcontrol.wait`. That is safe only if the preceding GEMM2 triggers its dependents after its own wait. The graph-mode check saw no mismatch; a wait-first variant measured about 0.08 ms/step slower at concurrency 1 (on a different node) with no accuracy difference.
- Moving the prenorm GEMM + Sinkhorn onto the critical path instead of the side stream was measured: +17–20 µs per decoder layer at M = 1–32, with the mixes inline or with TileLang’s fused post + prenorm GEMM. So the side stream stays.

AI assistance (Claude Code) was used for this change, including the kernels, the tests and the evaluations above.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---

## comments

**mergify** (2026-09-25T07:46:34Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @zyongye.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**zyongye** (2026-09-25T23:53:40Z):

/ci run

**github-actions** (2026-09-25T23:54:00Z):

✅ Triggered [Buildkite CI #91351](https://buildkite.com/vllm/ci/builds/91351) for commit `25b86ba6aa63`.

**zyongye** (2026-09-26T07:17:11Z):

/ci run

**github-actions** (2026-09-26T07:17:30Z):

✅ Triggered [Buildkite CI #91377](https://buildkite.com/vllm/ci/builds/91377) for commit `e958ba6296fc`.

