---
title: '57643'
pageLayout: default
slug: thoughts/inference/vllm-prs/57643
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/57643.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#57643](https://github.com/aarnphm/aarnphm.github.io/issues/57643) \[Perf\]\[DSV4.1\] Fuse TP all-reduce with mHC input preparation

[vllm-project/vllm#57643](https://github.com/vllm-project/vllm/pull/57643) · WoosukKwon · MERGED · created 2026-09-18T23:04:55Z · merged 2026-09-21T03:28:07Z

## Purpose

Fuse TP all-reduce, mHC post mixing, delayed collapse, and RMSNorm at DSV4.1 decoder boundaries. This replaces three launches with one, preserves the BF16 boundaries, and retains coefficient overlap. The kernel reuses the existing MNNVL workspace without an additional persistent communication allocation.

Enabled by default for eligible SM100 TP4 forwards with 1–16 tokens, including speculative decoding. The selector requires hidden size 5120, hc_mult=4, EP disabled, and a multicast workspace; it does not restrict DP/PP size, LoRA, or attention backend. The existing overlap-support checks still apply, including disabling overlap for ubatching. Larger batches and graph boundaries use the existing operations. Sixteen is the largest size with a validated gain, not a measured crossover.

Builds on [#57603](https://github.com/aarnphm/aarnphm.github.io/issues/57603), now merged into main, which overlaps coefficient computation but does not fuse collectives. This differs from [#51244](https://github.com/aarnphm/aarnphm.github.io/issues/51244)’s terminal mHC/head fusion. Open-PR searches for `mhc allreduce` and `mhc fusion` found no duplicate of this decoder-boundary collective fusion.

The independent startup warmup fixes have been extracted: DeepGEMM mHC projection coverage is in [#57843](https://github.com/aarnphm/aarnphm.github.io/issues/57843), and cached FlashInfer MXFP8 tactics are in [#57842](https://github.com/aarnphm/aarnphm.github.io/issues/57842). Neither is included in this fusion PR.

## Communication choice and token cutoff

Keep multicast and the 16-token cutoff. A controlled GB200 TP4 experiment replaced multicast stores with explicit stores to each of the four peers, retaining the same math, Lamport synchronization, cleanup, and PDL. Push did not consistently improve performance:

| Tokens | Multicast kernel | Push kernel | Multicast model forward | Push model forward |
| ------ | ---------------: | ----------: | ----------------------: | -----------------: |
| 1      |        14.348 µs |   14.634 µs |                4.054 ms |           4.060 ms |
| 8      |        14.395 µs |   14.972 µs |                5.724 ms |           5.728 ms |
| 16     |        13.910 µs |   15.795 µs |                7.133 ms |           7.130 ms |

Kernel timings use cold-L2 CUDA-graph replay, five samples of 50 replays, and the maximum rank time. Model timings use the same synthetic-weight model instance, KV8704, identical tactics, and three samples of 20 replays; they exclude the output head and serving. Multicast and push model outputs were byte-identical on every rank at all nine tested sizes (1–128).

Extending fusion to 24, 32, 48, 64, 96, and 128 tokens regressed model-forward time in this screen, despite isolated kernel gains; coefficient overlap remained disabled above 16. Some nonstandard sizes used default tactics because exact cache buckets were absent. A follow-up using normal serving autotune buckets confirmed the gain at 1 and 16 tokens (4.667→4.060 ms and 7.783→7.127 ms). Near-boundary runs at 17 and 20 tokens failed the unfused baseline’s native-top-k correctness check, so no timings from those shapes are accepted. This supports retaining 16 conservatively; it does not establish the exact crossover. No new model accuracy evaluation was run for the experimental variants.

Multicast requires a supported NVLink/NVSwitch configuration and successful group memory setup; GPU generation alone does not establish support. Explicit peer stores could broaden support to Hopper-or-newer systems with suitable peer-memory access without multicast. That would require separate allocation/dispatch support and hardware validation: this PR remains SM100 TP4, and the experimental push variant reused the existing workspace. Push is a possible portability fallback, not a demonstrated GB200 performance improvement.

## Latest model-forward comparison against main

Measured candidate `dd63c1fa00` (runtime source identical to the subsequently merged PR head `7be123b000`) versus main `d2983f2f16` fetched at benchmark start, including merged overlap and the latest DSpark block-table fix in both arms. GB200 ×4, TP4/DP1, FlashInfer FP8 attention, MXFP4 sparse-logits indexer, auto MoE, KV8704, max model length 1M. Matched synthetic weights/KV and 210 cached tactics with normal hybrid token buckets. Five samples ×20 full-forward CUDA-graph replays after warmup, maximum rank time per sample; median reported below.

| Mode      | Batch | Component / tokens |     Main | Candidate | Reduction |
| --------- | ----: | ------------------ | -------: | --------: | --------: |
| No DSpark |     1 | main / 1           | 4.651 ms |  4.055 ms |     12.8% |
| No DSpark |     8 | main / 8           | 6.236 ms |  5.668 ms |      9.1% |
| No DSpark |    16 | main / 16          | 7.806 ms |  7.127 ms |      8.7% |
| DSpark    |     1 | main / 6           | 5.867 ms |  5.280 ms |     10.0% |
| DSpark    |     1 | draft / 5          | 0.353 ms |  0.344 ms |      2.4% |

DSpark uses fixed draft length5. These are model-forward measurements, excluding the output head, sampling, scheduling, HTTP and acceptance effects. The draft implementation is unchanged, so its small difference is not attributed directly to fusion. All accepted cases pass eager/graph and changed/restored-input replay checks with zero error. Initialization manifests and source/final tactic hashes match between arms.

An additional candidate DSpark B2 target case failed the harness native-top-k score check at layer2; no timing from that shape is accepted, and main B2 was not measured. This run does not establish B2 correctness or new model accuracy. Untimed setup permits compilation and strict JIT checking resumes before validation/timing; neither arm includes the extracted startup fixes [#57842](https://github.com/aarnphm/aarnphm.github.io/issues/57842)/[#57843](https://github.com/aarnphm/aarnphm.github.io/issues/57843). The comparison measures warmed execution.

## Test Result

GB200 ×4, DeepSeek-V4.1-Flash revision `dba1be0a40aa45a94ad051997016db3960a90277`, TP4/DP1, FlashInfer FP8 attention, MXFP4 sparse-logits indexer, auto MoE, no DSpark. Baseline disables only this fusion before model construction; it includes [#57603](https://github.com/aarnphm/aarnphm.github.io/issues/57603). Both arms use the same checkpoint, configuration and FlashInfer autotuning cache (all 168 entries reused, checksum unchanged).

The latest polling/RMSNorm follow-up loads peer payloads together and distributes norm partials for local shared-memory reads. PDL completion stays early for multi-token forwards and moves to the end for one token. Compared with the previous fused kernel, a same-instance synthetic KV8704 forward comparison measured B1 4.558/4.588→4.044/4.044 ms, B8 6.014/6.012→5.969/5.979 ms, and B16 7.787/7.782→7.794/7.822 ms. B1 baseline varied between runs (4.184 ms in the prior controlled run), so the largest paired percentage is not a robust gain estimate. B16 is 0.1–0.5% slower in the final screen. Output bytes match on every rank for all three sizes; the final collective test also covers signed-zero sentinel collisions.

The serving and GSM8K measurements below are from the initial fused implementation (`fa59bb3883`). They have not been rerun after this output-equivalent follow-up.

8K input / 1K output serving, concurrency 1, three requests per run, forced output length:

| Run                           | Mean TPOT | Median ITL |   P99 ITL |
| ----------------------------- | --------: | ---------: | --------: |
| Baseline                      |  4.947 ms |   4.951 ms |  5.034 ms |
| Candidate, initial            |  5.361 ms |   4.510 ms | 16.240 ms |
| Candidate, repeat after GSM8K |  4.578 ms |   4.506 ms |  7.676 ms |

The warmed repeat reduces mean TPOT by 7.5%, but the initial candidate had stalls and tail latency remains higher. These short serving runs do not establish a tail-latency improvement. All requests completed, with 8192 input and 1024 output tokens each.

A separate fixed-forward screen measured B8 6.565→6.028 ms (−8.2%) and B16 8.421→7.785 ms (−7.6%). B32 exercises fallback: 11.382→11.353 ms with identical output. These are three-sample synthetic-weight/cache measurements at KV8704, excluding the output head and serving.

Full GSM8K, 1319 questions, five-shot chat, thinking disabled, greedy generation, seed 123, maximum 1024 generated tokens, concurrency 8:

| Metric              |           Baseline |          Candidate |
| ------------------- | -----------------: | -----------------: |
| Flexible extraction | 1267/1319 (96.06%) | 1275/1319 (96.66%) |
| Strict match        |             79.08% |             79.98% |

Prompt and target hashes match for every question. Floating-point ordering changes outputs; this is not bitwise model equivalence. The paired results have 11 candidate-only correct answers and 3 baseline-only correct answers.

- Four-GPU collective test passed: changing sizes, all-gather interleaving in the shared workspace, changed-input CUDA-graph replay.
- mHC regression selection: 137 passed.
- 77 saved real operator inputs: maximum relative RMS 2.3e-5 for residuals and 1.5e-4 for normalized inputs.
- CMake `_C_stable_libtorch` build and pre-commit checks passed.

Serving uses max model length [`1048576`](https://github.com/aarnphm/aarnphm.github.io/commit/1048576), max sequences 8, max batched tokens 8192, 10 GiB KV cache, prefix caching disabled, Engram CPU offload, and `--compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY","cudagraph_capture_sizes":[1,2,4,8]}'`. The sequence cap is a validation setting, not a recipe recommendation.

AI assistance was used for implementation and validation. The human submitter confirmed review and relevant tests for the original submission; the warmup follow-up validation is reported separately below.

## Warmup validation

The sparse-candidate alignment fix and its tests were extracted into [#57667](https://github.com/aarnphm/aarnphm.github.io/issues/57667). The validation below was run on combined commit `2219a281f8`, before the split, and includes that fix.

The missing mHC specialization was DeepGEMM split=4, not a TileLang epilogue. Warmup now covers splits 1/4/16 for both projection widths on GB200. A fresh-cache test compiled six variants, then successfully checked 26 runtime shapes with further compiler invocations forbidden.

A fresh real-weight TP4 server with `enable_jit_warmup=true` and `--jit-monitor-mode error --jit-monitor-verbose` completed 23 requests (736 generated tokens), spanning 3–8192 prompt tokens and mixed batches of up to eight requests. All four ranks enabled the monitor before requests: zero runtime Triton/TileLang/CuTeDSL JIT events, and no DeepGEMM compiler commands in this run. The 66 focused tests below and pre-commit checks passed.

The original serving measurements above explicitly disabled JIT warmup. These fixes close reproduced intermediate-prefill compilation gaps; they do not establish the cause of the original decode tails or a new steady-state speedup. GSM8K was not repeated for this startup/specialization follow-up, which leaves model arithmetic unchanged.

## Test Plan



```bash
RAY_ADDRESS=local .venv/bin/python -m pytest \
  tests/distributed/test_custom_all_reduce.py -k all_reduce_mhc -q -rs
.venv/bin/python -m pytest tests/kernels/test_mhc_kernels.py \
  -k 'deepseek_v41 or delayed_custom_op' -q -rs
```

Combined warmup follow-up at `2219a281f8` (66 passed, 2 deselected):



```bash
.venv/bin/python -m pytest tests/kernels/test_mhc_jit_warmup.py \
  tests/model_executor/test_flashinfer_autotune_warmup.py \
  tests/model_executor/layers/test_mla_short_prefill_indexer.py \
  -k 'not sparse_mqa_logits_prefill_matches_dense_masked and not deep_select' -q -rs
```

## Rebase validation

Rebased onto main `9b49f92344` (merged [#57603](https://github.com/aarnphm/aarnphm.github.io/issues/57603)). Retained the simplified overlap helper and existing fusion eligibility, multicast transport, and 16-token cutoff. The fused CUDA kernel and dispatcher are unchanged. Dropped the obsolete extraction commit so the sparse-indexer alignment fix already on main remains intact.

At `75e3082ab1`, all pre-commit hooks passed and the focused GB200 GPU suite passed (176 passed, 135 deselected):



```bash
RAY_ADDRESS=local .venv/bin/python -m pytest \
  tests/distributed/test_custom_all_reduce.py tests/kernels/test_mhc_kernels.py \
  -k 'all_reduce_mhc or deepseek_v41 or delayed_custom_op' -q -rs
```

Serving benchmarks and GSM8K were not repeated for this rebase; the earlier results above retain their stated scope.

## Broader dispatch validation

Removed benchmark-only restrictions on DP/PP size, attention backend, LoRA, and speculative decoding. The TP4 group, fixed kernel dimensions, EP exclusion, and multicast workspace checks remain. The kernel and 16-token cutoff are unchanged. The distributed replay test now includes 6/12-token fixed verification and shrinking/growing token counts.

The completed TP4 DSpark comparison (2026-09-19, before the rebase) used draft length five, FlashInfer FP8 attention, the same checkpoint and matched 210-entry tactics. Baseline retained overlap and disabled only fusion. Both arms scored 125/128 on a matched GSM8K subset; mean acceptance lengths were 3.739 and 3.730. All ten 8K-input/1K-output serving requests per arm succeeded. Mean TPOT was 1.451→1.247 ms, but random-prompt acceptance also changed (5.241→5.803), so this is not an isolated fusion speedup.

That validation used identical pinned-tactic and startup warmup workarounds in both arms. It is a subset accuracy check for fixed DSpark, not validation of every admitted backend, topology, LoRA configuration, or speculative method. Adaptive shape changes are covered at the distributed-kernel level. No new full GSM8K evaluation was run for this dispatch-only follow-up.

Follow-up `2b4ce61608`: 182 GPU/dispatch tests passed, 135 deselected, using the focused command above on GB200 ×4; all pre-commit hooks passed.

## Cached-tactic warmup extraction

Removed `_warmup_flashinfer_mxfp8_tactics`, its call site, and its unit test in `f6232a4f7f`. The independent two-file fix is in [#57842](https://github.com/aarnphm/aarnphm.github.io/issues/57842) at `622e20e1ee`. The mHC projection-split warmup was subsequently extracted to [#57843](https://github.com/aarnphm/aarnphm.github.io/issues/57843). The remaining autotune suite passed (4 tests), and pre-commit passed. Historical validation above that included the cached-tactic warmup still depends on the extracted fix; this PR alone does not close that runtime compilation gap.

## DeepGEMM projection warmup extraction

Removed `MHCPrenormGemmWarmup`, its model registration, and tests in `ef850bb5c3`; the standalone fix is [#57843](https://github.com/aarnphm/aarnphm.github.io/issues/57843). Historical combined validation above included both startup fixes ([#57842](https://github.com/aarnphm/aarnphm.github.io/issues/57842) and [#57843](https://github.com/aarnphm/aarnphm.github.io/issues/57843)). The fusion kernel and dispatch are unchanged by the extraction.

## Dispatch organization

At `70f0cb7be4`, `mhc.py` owns the shared `mhc_shifted_post_pre` dispatcher and `supports_mhc_all_reduce` eligibility check. `mega_mhc.py` contains only the DeepGEMM implementation and its support check. This is a code move with unchanged arithmetic, eligibility, and 16-token cutoff. Pre-commit passed; the GPU regression command below passed 192 tests (125 deselected). No accuracy/performance rerun for this structural change.



```bash
RAY_ADDRESS=local .venv/bin/python -m pytest \
  tests/distributed/test_custom_all_reduce.py tests/kernels/test_mhc_kernels.py \
  -k 'all_reduce_mhc or deepseek_v41 or delayed_custom_op or deep_gemm_mega_mhc' -q -rs
```

---

## comments

**mergify** (2026-09-20T23:49:34Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @WoosukKwon.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**WoosukKwon** (2026-09-21T01:13:10Z):

/ci run

**github-actions** (2026-09-21T01:13:27Z):

❌ This PR is 1 commit behind upstream `main`. Your branch must contain every commit currently on upstream `main`. No new CI build was started. Merge or rebase onto the latest `main`, then rerun `/ci run`. To test this branch at your own risk, use `/ci run --allow-stale`.

<!-- vllm-ci-command:5754133622 -->

**WoosukKwon** (2026-09-21T01:21:50Z):

/ci run

**github-actions** (2026-09-21T01:22:10Z):

✅ Triggered [Buildkite CI #90112](https://buildkite.com/vllm/ci/builds/90112) for commit `7be123b00037`.

