---
title: '57204'
pageLayout: default
slug: thoughts/inference/vllm-prs/57204
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/57204.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#57204](https://github.com/aarnphm/aarnphm.github.io/issues/57204) \[Perf\]\[DSV4.1\] Remove MegaMoE padding and shared padding workaround

[vllm-project/vllm#57204](https://github.com/vllm-project/vllm/pull/57204) · WoosukKwon · MERGED · created 2026-09-16T16:18:27Z · merged 2026-09-16T19:09:30Z

## Purpose

Remove unnecessary MegaMoE intermediate-width padding for DeepSeek-V4.1-Flash and revert the production workaround added in [#56568](https://github.com/aarnphm/aarnphm.github.io/issues/56568). Routed and shared experts both retain the checkpoint width of 2304, so native shared-expert fusion remains enabled without padding either to 2560.

The bundled DeepGEMM uses `layout::Data(..., false)` for activation-scale rows in `include/deep_gemm/layout/mega_moe.cuh`; the old 16-byte TMA-row alignment rationale no longer applies. The existing runtime check still requires intermediate dimensions divisible by 128.

This removes 10% of the intermediate dimension used by the padded expert GEMMs and avoids the separate padded shared-weight copies introduced by [#56568](https://github.com/aarnphm/aarnphm.github.io/issues/56568). Fusion keeps its existing eligibility checks and disable flag. A bare revert of [#56568](https://github.com/aarnphm/aarnphm.github.io/issues/56568) would leave routed padding in place and disable V4.1 shared fusion again, so this change removes both together.

Retain the useful regression coverage from [#56568](https://github.com/aarnphm/aarnphm.github.io/issues/56568), now checking exact checkpoint-shaped weight/scale preservation, native-width shared fusion, buffer reuse, and repeated finalization.

Related: [#56217](https://github.com/aarnphm/aarnphm.github.io/issues/56217). Duplicate checks covered its issue/comments, open PRs mentioning [#56217](https://github.com/aarnphm/aarnphm.github.io/issues/56217)/[#56568](https://github.com/aarnphm/aarnphm.github.io/issues/56568), and MegaMoE/shared-expert/padding searches. [#56568](https://github.com/aarnphm/aarnphm.github.io/issues/56568) is already merged; this replaces its padding workaround. [#54049](https://github.com/aarnphm/aarnphm.github.io/issues/54049) targets FlashInfer CuteDSL, [#55275](https://github.com/aarnphm/aarnphm.github.io/issues/55275) adds GLM adapters, [#40833](https://github.com/aarnphm/aarnphm.github.io/issues/40833) integrates a general fused-MoE backend, and [#56697](https://github.com/aarnphm/aarnphm.github.io/issues/56697) targets SM90 Marlin. None removes this native DeepGEMM model-path padding.

## Test Plan



```bash
.venv/bin/python -m pytest tests/models/test_deepseek_v4_mega_moe.py -q
pre-commit run --files vllm/models/deepseek_v4/nvidia/model.py tests/models/test_deepseek_v4_mega_moe.py
pre-commit run mypy-3.12 --files vllm/models/deepseek_v4/nvidia/model.py tests/models/test_deepseek_v4_mega_moe.py --hook-stage manual
```

Also compare this implementation directly with upstream `42919b49c5` on four GB200 GPUs using identical random expert weights, inputs, routes, and routing weights: hidden size 5120, intermediate width 2304 versus 2560, 384 experts, top-6, token counts 1/8/32/128/1024/2048, shared fusion both disabled and enabled. For the reference, upstream itself pads the native checkpoint-shaped shared weights.

## Test Result

Focused suite: **30 passed** on GB200.
Four-GPU operator comparison against upstream `42919b49c5`: **48/48 comparisons have exactly zero output error**, including all shared-fusion cases. Shared-expert outputs have nonzero contribution. Candidate commit: `21cda373e8`.
Applicable pre-commit hooks, including mypy 3.10, and explicit mypy 3.12 passed.

Previously completed real-weight model evaluation used this exact production model file with `deepseek-ai/DeepSeek-V4.1-Flash`, revision `dba1be0a40aa45a94ad051997016db3960a90277`, TP4/EP4. Full GSM8K with lm-eval 0.4.12, five-shot multi-turn chat, thinking disabled, greedy generation, 1024 output tokens, seed 123:

| Configuration                                          | Correct / 1319 |         Accuracy |
| ------------------------------------------------------ | -------------: | ---------------: |
| Auto-MoE/FlashMLA FP8-indexer baseline, three runs     |      1270–1274 | 96.2851–96.5883% |
| Native-width MegaMoE/FlashMLA FP8 indexer              |           1271 |         96.3609% |
| Combined optimized configuration, MXFP4 sparse indexer |           1266 |         95.9818% |

These are prior model-path validation results, not a newly measured isolated comparison against current upstream with [#56568](https://github.com/aarnphm/aarnphm.github.io/issues/56568). The combined configuration includes separate attention/indexer optimizations. No new end-to-end speedup over [#56568](https://github.com/aarnphm/aarnphm.github.io/issues/56568) is claimed here.

## AI assistance

Codex assisted with implementation, validation, and this description. Woosuk Kwon authorized submission after the prepared diff and validation results were presented for review.

---

## comments

**WoosukKwon** (2026-09-16T16:19:32Z):

/ci run

**github-actions** (2026-09-16T16:19:56Z):

✅ Triggered [Buildkite CI #89372](https://buildkite.com/vllm/ci/builds/89372) for commit `21cda373e87b`.

**WoosukKwon** (2026-09-16T17:15:33Z):

@gcanlin No worries at all! Actually it was me who added the padding to the routed experts in the first place (because that was before the DeepGeMM update). I should have tracked this issue more closely 😓

