---
title: '56962'
pageLayout: default
slug: thoughts/inference/vllm-prs/56962
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/56962.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#56962](https://github.com/aarnphm/aarnphm.github.io/issues/56962) \[Perf\]\[Kernel\] Integrate Mega-mHC from DeepGEMM for DeepSeek V4.1 (reopen of [#56255](https://github.com/aarnphm/aarnphm.github.io/issues/56255))

[vllm-project/vllm#56962](https://github.com/vllm-project/vllm/pull/56962) · Juntian777 · MERGED · created 2026-09-15T06:44:50Z · merged 2026-09-15T11:01:07Z

## Summary

- Reopen [vLLM #56255](https://github.com/vllm-project/vllm/pull/56255) (closed unmerged), rebased on the [#56876](https://github.com/vllm-project/vllm/pull/56876) base (`1b5a7abb58`) with the model path retargeted to `deepseek_v41`.
- Keep the original implementation commit separate from the follow-up adaptation; its elfutils dependency is already in the base.
- Preserve the current TileLang fused path for auxiliary capture and unsupported inputs, and preserve Engram injection order.
- Pin the [#56876](https://github.com/aarnphm/aarnphm.github.io/issues/56876) vLLM DeepGEMM fork at `ad1f1726` (2.8.0), including its SM90/SM120 support and API fixes.

## Why this is not a duplicate

[#56255](https://github.com/aarnphm/aarnphm.github.io/issues/56255) was closed without merging; this PR reopens that work. Open-PR searches show no competing Mega-mHC integration. [#56611](https://github.com/vllm-project/vllm/pull/56611) fuses TileLang statistics with attention/FFN instead; this change uses DeepGEMM Mega-mHC fusion. Dependency changes intentionally reuse [#56876](https://github.com/aarnphm/aarnphm.github.io/issues/56876) rather than re-opening them.

## Benchmarks

Single GB200, H=5120 / 7168, `hc_mult=4`, BF16, 20 Sinkhorn iterations. Full post + delayed-pre + RMSNorm chain. CUPTI full-stage spans over CUDA graph replays, cold L2, 10 warmup + 100 samples per variant in forward/reverse (ABCCBA) order. Setup, JIT, and host allocation excluded; CPU launch overhead is outside the measured spans.

|    H | Decode tokens | Split (µs) | Current TileLang fused (µs) | Mega-mHC (µs) | vs current |
| ---: | ------------: | ---------: | --------------------------: | ------------: | ---------: |
| 5120 |             1 |      16.29 |                       13.30 |         11.68 |      1.14× |
| 5120 |             8 |      17.44 |                       13.31 |         11.62 |      1.15× |
| 5120 |            64 |      18.21 |                       18.30 |         12.64 |      1.45× |
| 5120 |           256 |      20.90 |                       21.14 |         15.90 |      1.33× |
| 7168 |             1 |      20.67 |                       17.02 |         12.86 |      1.32× |
| 7168 |             8 |      20.61 |                       15.71 |         12.13 |      1.30× |
| 7168 |            64 |      22.27 |                       22.37 |         14.85 |      1.51× |
| 7168 |           256 |      25.41 |                       25.47 |         17.28 |      1.47× |

### E2E Benchmarks

GB200 TP4, official DeepSeek-V4.1-Flash, `fp8_ds_mla`, no speculative decoding, `max-model-len=40960`. Same local source (`e7fcd569dd`) and dependencies; TileLang uses the existing fused-op worker hook, and [#56611](https://github.com/aarnphm/aarnphm.github.io/issues/56611) Overlap is ported onto this base with TP4 enabled.

`vllm bench serve`: exactly 8192/32768 input + 1024 output tokens, seed 0, temperature 0, one warmup, ignore EOS, prefix caching disabled. C1: 20 requests/case; C32: 64. All **504 requests** completed with exact token counts.

Latencies are in ms. **Δ is relative to TileLang**, shown as mean TPOT / output throughput; negative TPOT and positive throughput are better.

**Concurrency 1:**

| Input | Variant  | TPOT mean / p50 | TTFT p50 | TTFT p99 | E2E p50 | Output tok/s | Δ TPOT / tok/s |
| ----: | -------- | --------------: | -------: | -------: | ------: | -----------: | -------------: |
|    8k | TileLang |   7.466 / 7.464 |    211.7 |    227.6 |  7845.1 |        130.5 |              — |
|    8k | Mega-mHC |   7.352 / 7.351 |    190.1 |    192.9 |  7710.3 |        132.8 |  -1.5% / +1.8% |
|    8k | Overlap  |   7.355 / 7.351 |    204.7 |    215.1 |  7724.5 |        132.5 |  -1.5% / +1.6% |
|   32k | TileLang |   7.503 / 7.502 |    788.2 |    807.3 |  8462.8 |        121.0 |              — |
|   32k | Mega-mHC |   7.396 / 7.394 |    715.0 |    737.8 |  8284.4 |        123.6 |  -1.4% / +2.2% |
|   32k | Overlap  |   7.394 / 7.394 |    769.2 |    783.6 |  8336.5 |        122.9 |  -1.5% / +1.5% |

**Concurrency 32:**

| Input | Variant  | TPOT mean / p50 | TTFT p50 | TTFT p99 | E2E p50 | Output tok/s | Δ TPOT / tok/s |
| ----: | -------- | --------------: | -------: | -------: | ------: | -----------: | -------------: |
|    8k | TileLang | 13.565 / 14.518 |   1016.7 |   5555.9 | 15799.0 |       2055.2 |              — |
|    8k | Mega-mHC | 13.659 / 13.692 |    913.4 |   4948.5 | 15740.2 |       2069.2 |  +0.7% / +0.7% |
|    8k | Overlap  | 14.037 / 14.394 |   1007.7 |   5517.2 | 16318.7 |       1995.2 |  +3.5% / -2.9% |
|   32k | TileLang | 23.500 / 27.256 |   2947.8 |  21745.9 | 31014.7 |       1039.9 |              — |
|   32k | Mega-mHC | 22.827 / 26.943 |   2732.4 |  20366.1 | 29700.7 |       1083.7 |  -2.9% / +4.2% |
|   32k | Overlap  | 23.842 / 28.315 |   2907.7 |  21991.1 | 31213.0 |       1032.6 |  +1.5% / -0.7% |

**Conclusion:** Mega-mHC and Overlap have essentially equal C1 TPOT; Mega has lower TTFT. At C32, Mega delivers **3.7% / 5.0% more output tok/s than Overlap** at 8k / 32k.

## Tests

`.venv/bin/python -m pytest tests/kernels/test_mhc_kernels.py -q -x`: **174 passed, 32 skipped** — covers empty/decode/prefill and changed-input CUDA graph replay. FP8 DeepGEMM tests: **100 passed, 25 skipped**; matching-extension qnorm/RoPE tests: **6 passed**. Pre-commit checks passed.

| Task         | Configuration                                                                        |              Score | Request errors | Output-limit hits |
| ------------ | ------------------------------------------------------------------------------------ | -----------------: | -------------: | ----------------: |
| GSM8K        | 1319 problems, 5-shot, greedy, max output 4096                                       | 1225/1319 (92.87%) |              0 |                 1 |
| GPQA Diamond | 198 problems, 0-shot, default high thinking, temp 1.0, top_p 0.95, max output 262144 |   179/198 (90.40%) |              0 |                 0 |

Both accuracy evaluations used Mega-mHC with the official `deepseek-ai/DeepSeek-V4.1-Flash` checkpoint, on exclusive GB200 nodes with TP4+EP, `fp8_ds_mla`, DeepGEMM MegaMoE, CPU-offloaded Engram, no speculative decoding, and prefix caching disabled. One pass run at concurrency 32, request seed 0 and dataset/few-shot seed 1234. GSM8K: 1319 problems, 5-shot, completions endpoint, greedy (`temperature=0`, `top_p=1`), max output 4096 and server context 8192; strict-match and flexible-extract both scored **1225/1319 (92.87%)**, with one output-limit hit. GPQA Diamond: 198 problems, 0-shot, chat completions, `think=true`, default `high` reasoning effort (not explicitly overridden), `temperature=1`, `top_p=0.95`, max output 262144 and server context [`1048576`](https://github.com/aarnphm/aarnphm.github.io/commit/1048576); **179/198 (90.40%)**, with no output-limit hits. Both had zero request errors; server max batched tokens was 8192, max sequences 256.

## AI assistance

This change was developed with AI assistance (OpenAI Codex); all changes were reviewed and tested by the human submitter.

---

## comments

**github-actions** (2026-09-15T10:17:41Z):

✅ @Juntian777, CI is now available for this PR.

- `/ci run` starts upstream CI; `/amd-ci run` starts AMD CI only.
- Your branch must contain every commit currently on its upstream target branch. Merge or rebase onto the latest target branch, then rerun the command. Append `--allow-stale` to a run command to test an outdated branch at your own risk.
- `/ci retry` retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
- `/amd-ci retry` retries failed jobs in AMD CI for the current PR head. Use `/amd-ci run` when the current head has no AMD CI build.
- `/ci cancel` cancels scheduled or running CI builds for this PR branch; `/amd-ci cancel` does the same for AMD CI only.

<!-- vllm-ci-authorized -->

**ywang96** (2026-09-15T10:18:19Z):

/ci run

**github-actions** (2026-09-15T10:18:37Z):

✅ Triggered [Buildkite CI #89077](https://buildkite.com/vllm/ci/builds/89077) for commit `91368b7e2330`.

