---
title: '56266'
pageLayout: default
slug: thoughts/inference/vllm-prs/56266
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/56266.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#56266](https://github.com/aarnphm/aarnphm.github.io/issues/56266) \[DSv4.1\] Integrate Mega-Gate from DeepGEMM

[vllm-project/vllm#56266](https://github.com/vllm-project/vllm/pull/56266) · gau-nernst · MERGED · created 2026-09-10T12:33:11Z · merged 2026-09-17T19:06:37Z

## Summary

Integrate DeepGEMM Mega-Gate into the DeepSeek V4/V4.1 DeepGEMM MegaMoE path, fusing the gate GEMM and expert selection. Prepare routing metadata once per model forward and reuse it across layers. Preserve hash routing for V4, image routing bias for V4.1, draft-layer expert counts, MTP warmup, and decoder auxiliary-state capture.

Two integration fixes are included:

- Let CMake FetchContent update an existing DeepGEMM checkout when its pinned revision changes. Previously, finding `csrc/python_api.cpp` skipped the update entirely and could rebuild 2.6.1 without `bf16_mega_gate` despite the new pin.
- ~~Clear padded hidden-state rows in place before Mega-Gate. NaN padding can trigger the kernel’s finite-score assertion, later surfacing as a CUDA launch failure or NCCL error. `masked_fill_` avoids allocating a separate hidden-state tensor; valid rows are unchanged. Regression cases cover NaN padding for both main/draft and text/image routing.~~ Removed, fixed by [vllm-project/DeepGEMM#11](https://github.com/vllm-project/DeepGEMM/pull/11)

Update (17 Sep 2026)

- Since Mega-Gate underperforms for M<span>&lArr;</span>16 (as LL-GEMM is very efficient), we only dispatch Mega-Gate for M>16

## Accuracy: full GSM8K

The final in-place implementation was evaluated locally through Vigil on **4× GB200**, using real `deepseek-ai/DeepSeek-V4.1-Flash` weights, the current repository checkout, CUDA 13.1, and PyTorch 2.13.0. No engine overrides beyond TP4, EP, and the MegaMoE backend were supplied; CUDA Graphs remained enabled.

| Metric            | Result       |
| ----------------- | ------------ |
| Questions         | 1319         |
| Correct           | **1194**     |
| Accuracy          | **90.5231%** |
| Invalid responses | **0**        |
| API failures      | **0**        |
| Worker errors     | **0**        |
| Evaluation time   | 161.79 s     |

## Kernel benchmark: DeepSeek V4.1 Flash Gate

Measured on a **single NVIDIA GB300 (SM103)** with BF16 inputs `[M, 5120]`, BF16 gate weights `[384, 5120]`, **top-k=6**, `sqrtsoftplus`, normalization, and routed scaling factor **1.5**. **M is the number of token rows processed on one GPU**, not serving request concurrency.

The baseline is `GateLinear + fused_topk_bias`, using its normal shape-dependent dispatch. Mega-Gate fuses the gate GEMM and expert selection. The updated DeepGEMM pin allows non-finite padding scores, eliminating the separate hidden-state padding clear; existing input staging discards padded routes.

**Mega-Gate achieves 1.18–1.31× routing speedup for M ≥ 32, with no meaningful speedup for M ≤ 16.** Speedup is baseline latency divided by Mega-Gate latency.

| Tokens M | GateLinear + top-k (μs) | Mega-Gate (μs) | Speedup |
| -------: | ----------------------: | -------------: | ------: |
|        1 |                   6.640 |          8.096 |  0.820× |
|        2 |                   6.848 |          8.032 |  0.853× |
|        4 |                   6.752 |          8.032 |  0.841× |
|        8 |                   8.064 |          8.224 |  0.981× |
|       16 |                   8.256 |          8.256 |  1.000× |
|       32 |                  10.112 |          8.304 |  1.218× |
|       64 |                  10.353 |          8.448 |  1.225× |
|      128 |                  11.200 |          8.896 |  1.259× |
|      256 |                  11.488 |          9.696 |  1.185× |
|      512 |                  13.520 |         10.720 |  1.261× |
|     1024 |                  16.000 |         12.448 |  1.285× |
|     2048 |                  18.913 |         15.872 |  1.192× |
|     4096 |                  27.729 |         22.944 |  1.209× |
|     8192 |                  46.752 |         35.745 |  1.308× |

**Method:** FlashInfer CUPTI timing of CUDA-graph replays with cold L2, 25 warmups, three rounds with alternating implementation order, and 100 samples per round. Values are medians of the round medians. Allocation, compilation, graph capture, input generation, routing metadata preparation, expert input staging, expert execution, and communication are outside the timed region. These measurements do not establish an E2E serving speedup.

**Correctness:** Expert IDs matched exactly at every batch size; routing weights matched with `rtol=1e-3, atol=1e-4`. Synthetic inputs and separated correction biases avoid ambiguous top-k boundaries. An additional 48 CUDA-graph replay cases passed with NaN/Inf padding across text, image, and hash/image routing.

**Versions:** vLLM `ef50fde183`, DeepGEMM `40d109bf`, PyTorch `2.13.0+cu130`, CUDA toolkit `13.1`, FlashInfer `0.6.18`, and CUPTI Python `13.0.1`.

## Related work and AI assistance

This updates the existing Mega-Gate PR rather than opening another implementation. Open-PR searches for `56266 in:body` and `Mega-Gate` were checked before reopening. Dependency/API changes overlap intentionally with [#56876](https://github.com/aarnphm/aarnphm.github.io/issues/56876); that PR supplies the dependency update, while this PR adds the Mega-Gate routing integration. The upstream Mega-Gate implementation is tracked in [deepseek-ai/DeepGEMM#432](https://github.com/deepseek-ai/DeepGEMM/pull/432).

OpenAI Codex assisted with the port, dependency alignment, conflict resolution, build-cache and padding fixes, compilation, debugging, and validation. Dependency/API changes adapted from [#56876](https://github.com/aarnphm/aarnphm.github.io/issues/56876) are attributed in the commit history.

---

## comments

**ywang96** (2026-09-12T08:39:09Z):

## Review: Mega-Gate integration (correctness + style)

This PR replaces the `fused_topk_bias` router with DeepGEMM’s fused `bf16_mega_gate` kernel (gate GEMM + scoring + bias + top-k in one kernel) for all MegaMoE backends, deriving the extra routing metadata (image-token mask, hash-routing masks, safe hash input IDs) once per model forward and threading it through the layers as `MegaGateRoutingMetadata`. Overall this looks correct and carefully wired — **approve with comments**; findings below, with the merge-blocking items called out at the end.

I verified the wrapper against the pinned DeepGEMM commit (`39d8c4c`): positional/keyword argument order, `out=(topk_idx, topk_weights)` tuple order, int64/fp32 output dtypes, and the `image_bias`/`image_token_mask` co-presence assert all match `csrc/apis/mega_gate.hpp`. The kernel always emits normalized weights (`selected_weight / topk_sum * routed_scaling_factor` in the epilogue), so the new `use_mega_moe and not renormalize` init-time guard is correct and necessary. `ep_rank` is only consumed for EPLB duplicate selection / random routing (neither used here), so passing it is benign. SP handling is consistent in all four entry points (metadata derived from the already-sharded `input_ids`; MTP explicitly shards `routing_input_ids`), and draft-layer prefixes (`layers.{num_hidden_layers+i}`) guarantee draft MoEs are never hash-routed, so the hardcoded `has_hash_routing=False` in mtp/dspark is safe.

## Correctness findings

**Major**

1. **DeepGEMM pin bump silently drops SM120 and CUDA<12.9 support** — `cmake/external_projects/deepgemm.cmake:43-69`, `tools/install_deepgemm.sh:75`. The previous pin was explicitly chosen for SM120 (“nv_dev branch (SM120 support)”); the new one removes `12.0f/12.0a/12.1a` from `DEEPGEMM_SUPPORT_ARCHS` and raises the SM90 CUDA floor from 12.3 to 12.9. On SM120 builds the arch intersection is now empty, so DeepGEMM `_C` silently isn’t built. This branch carries SM120-specific DSv4 code (`DeepseekV4FlashInferSM120Attention`, `model.py:1158-1160`), so this is a regression on this branch, not just a future-merge concern — currently only acknowledged in a cmake comment. Please call it out in the PR description and/or gate it so SM120 builds fail loudly or fall back instead of shipping without DeepGEMM.

2. **FlashInfer mega-MoE backends now silently require the SM100-only DeepGEMM gate** — `vllm/config/kernel.py:160` (`MEGA_MOE_BACKENDS = {"deep_gemm_mega_moe"} | FLASHINFER_MOE_EP_BACKENDS`), `vllm/models/deepseek_v4/nvidia/model.py:1087`. Because `use_mega_moe` is true for the FI EP backends, their routing now flows through `bf16_mega_gate`, which hard-asserts `arch_major == 10` in C++. The experts-level `_check_runtime_supported` (`model.py:362-365`) covers the DeepGEMM experts path, but a stale DeepGEMM install surfaces only via the wrapper’s generic `RuntimeError` at first forward. An explicit SM100 capability check at `DeepseekV4MoE.__init__` when `use_mega_moe` would make this actionable.

**Minor**

3. **No Python-side validation in the `bf16_mega_gate` wrapper** — `vllm/utils/deep_gemm.py:462-510`. DeepGEMM requires bf16 contiguous `x`, `hidden % 256 == 0`, `num_routed_experts % 4 == 0`; violations surface as raw C++ `DG_HOST_ASSERT`s (this is why the test had to bump `hidden_size` 128→256). Note the existing experts guard checks `hidden % 128` (`model.py:366`) while the gate needs `% 256`. A one-line dtype/shape check would turn these into actionable errors.

4. **Vision+hash combined branch untested and fragile to config drift** — the two kernel tests cover (vision, no-hash) and (hash, no-vision) separately; the `torch.where(image_token_mask, 0, input_ids)` + `~image_token_mask` path (`model.py:476-480`) is never exercised. DeepGEMM asserts `image_bias.has_value() == image_token_mask.has_value()`, so any draft/target `vision_n_layers` inconsistency crashes deep in C++. Configs are consistent today (both sides read the same hf_config), but a unit test for this branch would lock that in.

5. **Test reference mask narrower than production** — `tests/models/test_deepseek_v4_mega_moe.py:113` treats only `IMAGE_SENTINEL_BASE_ID` and `IMAGE_PAD_ID` as image tokens; production masks all five sentinel ids. The test inputs only contain those two, so it passes, but `IMAGE`/`IMAGE_NEW_LINE`/`IMAGE_END` sentinels are never exercised.

**Nit**

6. `vllm/utils/deep_gemm.py:476-480` — the docstring sentence “Passing PyTorch-owned outputs to DeepGEMM makes their storage part of CUDA graph capture” is cryptic; the point is that torch-allocated `out` tensors get stable addresses across graph replays.

<details><summary>Verified OK (no action needed)</summary>

- Output tuple order: DeepGEMM returns `(topk_idx, topk_weights)`; the wrapper passes `out=(topk_idx, topk_weights)` and returns `(topk_weights, topk_idx)` per vLLM router convention. ✓
- `num_physical_topk == num_topk` with `use_shared_as_routed=False`, so the wrapper’s `torch.empty((T, K))` shapes match the C++ `check_tensor`. ✓
- EPLB mapping still happens downstream in `DeepseekV4MegaMoEExperts.forward` (`model.py:726-743`); the gate correctly returns logical IDs. ✓
- `ValueError`-before-`RuntimeError` ordering in `forward` keeps `test_deepseek_v4_moe_preserves_configured_hash_layers` passing. ✓
- `tid2eid` is int64 on the mega path (`model.py:888`), matching the kernel’s int64 `unmapped_topk_idx` requirement; `input_ids.to(torch.int64)` precedes SP sharding in both main models. ✓
- Metadata computation (`torch.where`, `ones_like`, comparisons, index_select) is CUDA-graph-capturable and computed once per forward, outside the layer loop. ✓
- AMD variants have their own model/dspark implementations — no missed callers of the changed forward signatures. ✓
- Submodule switch (fmt → deep_jit) matches the pinned commit’s `.gitmodules`; `install_deepgemm.sh` and cmake pins are in sync. ✓

</details>

## Style findings

**Minor**

1. **The same \~7-line metadata block is copy-pasted into five forward methods** — `v4/nvidia/model.py:1511-1519`, `v4/nvidia/dspark.py:211-219`, `v4/nvidia/mtp.py:184-195`, `v4_1/nvidia/model.py:618-626`, `v4_1/nvidia/dspark.py:206-214`. Each repeats the `mega_gate_metadata = None / if self.use_mega_moe: ...` scaffolding plus the vision predicate `IMAGE_SENTINEL_BASE_ID if getattr(self.config, "vision_n_layers", 0) > 0 else None` — a predicate the MoE class already encodes at `model.py:874-876`. Six copies of logic that must stay in sync with `bias_vl` creation; letting `prepare_mega_gate_routing_metadata` take the config (or a `_has_vision(config)` helper) collapses each call site to one line.

2. **`prepare_mega_gate_routing_metadata` re-implements an existing helper with a magic number** — `model.py:471-473` hardcodes `+ 5`; `vllm/models/deepseek_v4/common/mm_preprocess.py:73-77` already provides `image_sentinel_mask()` keyed to `len(IMAGE_SENTINEL_TOKEN_NAMES)`. The `5` will silently drift if the sentinel family changes. Similarly, `torch.where(image_token_mask, 0, input_ids)` (`model.py:477`) uses a magic `0` fallback id with no comment.

3. **No docstrings on the two new module-level helpers** — `MegaGateRoutingMetadata` (`model.py:97-101`) and `prepare_mega_gate_routing_metadata` (`model.py:104-124`). Neighboring helpers carry one-line docstrings (e.g., `image_sentinel_mask`). The non-obvious contract — text models get all-`None` fields; `hash_token_mask` doubles as the kernel’s `fix_routing_mask`; image tokens get id 0 as a safe hash-table index — is exactly what should be documented.

**Nit**

4. **Error-type/message inconsistency in the MoE forward** — adjacent pre-existing checks use `ValueError` with a `DeepSeek V4 ...` prefix (`model.py:1057`, `1061`); the new checks use `RuntimeError` without the prefix and without naming the parameter (`model.py:1066-1069`, `1077`, `1079`). Same caller-error category, two conventions.

5. **Function-local import of `bf16_mega_gate` in the per-layer forward** — `model.py:1071`. `vllm/utils/deep_gemm.py` has no circular-import risk; this exists so `monkeypatch.setattr("vllm.utils.deep_gemm.bf16_mega_gate", ...)` works in the test. Defensible, but it looks accidental in a hot path — add a one-line comment, or import at module level and patch `vllm.models.deepseek_v4.nvidia.model.bf16_mega_gate` in the test (patch-where-used).

6. **`vllm/utils/deep_gemm.py` watched only by the DSv4 kernels CI area** — `.buildkite/test_areas/kernels.yaml`. The module is shared infrastructure (fp8 GEMM, MQA-logits wrappers); it’s absent from `attention.yaml`/`quantization.yaml`/etc., so future edits to it trigger only DSv4 tests.

7. **cmake comment narrates the previous revision** — `cmake/external_projects/deepgemm.cmake:68-69` (“The previous nv_dev pin also supported 12.0f …”). Useful as a restore-note, but describes code that no longer exists; will go stale on the next re-pin.

8. **`moe_backend in MEGA_MOE_BACKENDS` open-coded in three more files** — `mtp.py:98`, `dspark.py:81`, `v4_1/dspark.py:83` (model.py already had three copies). The codebase has the named-predicate precedent `is_fi_moe_ep_backend` (`model.py:85`); an `is_mega_moe_backend()` next to `MEGA_MOE_BACKENDS` in `vllm/config/kernel.py` would read better.

9. **Bare “PR 432” reference in a code comment** — `model.py:1080`. Which repo? It’s [DeepGEMM#432](https://github.com/DeepGEMM/aarnphm.github.io/issues/432); the permanent comment should link it.

10. **Test-style nits** — new `test_deepseek_v4_mega_gate_hash_routing_correctness` has no docstring while neighbors have one-liners; `tests/kernels/test_mhc_kernels.py:112` absorbs the new kwarg via `**kwargs` instead of naming `mega_gate_metadata=None` (hides future signature drift); the in-body `pytest.skip("DeepGEMM Mega Gate requires SM100")` could be `@pytest.mark.skipif` decorators matching the module’s `pytestmark` style.

**Verified clean (no finding):** import ordering satisfies ruff isort (`I`) in all touched files, and the relative-vs-absolute split (`..common.mm_preprocess` in v4.1 vs absolute in v4) matches each file’s pre-existing convention. No dead code from the removed `fused_topk_bias` import (`image_sentinel_lo` still feeds `_init_fused_moe_experts`, `hash_indices_dtype` still used for the `tid2eid` parameter). Type annotations complete and in file style; no prints, bare asserts, or commented-out code added; `_lazy_init` global/early-return plumbing follows the file’s pattern exactly.

## Merge guidance

Blocking for me: **Major 1** (the SM120/CUDA<12.9 DeepGEMM regression needs to be either fixed, loudly gated, or at minimum documented in the PR description, since this branch ships SM120 DSv4 code) and **Major 2** (an init-time SM100 capability check so FI-backend users don’t hit a raw C++ assert). Everything else — including the test-coverage gaps (Minor 4/5) and all style items — is follow-up or optional polish.

Nice work on the metadata threading; the once-per-forward design and the SP sharding consistency are clean.

**ivanusto** (2026-09-14T03:41:46Z):

Thanks for the Mega-Gate integration. One SM12x note on the pin change, from 2x DGX Spark (GB10, sm_121), where DeepSeek-V4-Flash currently serves on the `nv_dev` pin.

This PR moves the pin to DeepGEMM `main` and drops `12.0f`, `12.0a` and `12.1a` from `DEEPGEMM_SUPPORT_ARCHS`. DeepGEMM `main` has no SM12x sources, and `nv_dev` (latest commit `572557e`, 09-08) is 108 commits ahead and 1 behind `main`, so it has not been rebased onto the 26/09 release yet.

Unlike [#56255](https://github.com/aarnphm/aarnphm.github.io/issues/56255), this PR leaves `support_deep_gemm()` returning True for the whole 120 family. So on sm_121 DeepGEMM would still be reported as supported while no SM12x kernels are built. I have not built this PR to see exactly how that fails, but on current `main` several DSv4 paths call DeepGEMM on GB10: `mhc_pre_broadcast_tilelang` (which passes `use_tilelang_fallback=False`), `deep_gemm_fp8_o_proj` via `fp8_einsum` (no platform check), and the DeepGEMM FP8 block linear kernel that auto selects on sm_121.

Both this PR and [#56255](https://github.com/aarnphm/aarnphm.github.io/issues/56255) rewrite the same pin line, so whichever lands first sets SM12x’s fate for the other. Could SM12x keep a working path until `nv_dev` is rebased, for example by holding the arch removal or keeping a separate pin for 12.x, as the cmake comment already anticipates? Happy to test on 2x GB10.

**mergify** (2026-09-15T08:08:27Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @gau-nernst.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**mergify** (2026-09-15T12:10:04Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @gau-nernst.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**ZJY0516** (2026-09-17T16:34:29Z):

/ci run —allow-stale

**github-actions** (2026-09-17T16:34:47Z):

✅ Triggered [Buildkite CI #89670](https://buildkite.com/vllm/ci/builds/89670) for commit `4ecab30649ca`.

⚠️ This PR is 37 commits behind upstream `main`. Running CI at your own risk because `--allow-stale` was requested; outdated CI configuration may cause failures. Before merging, merge or rebase onto the latest `main`, then rerun `/ci run` on the latest PR commit.

