---
title: '56214'
pageLayout: default
slug: thoughts/inference/vllm-prs/56214
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/56214.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#56214](https://github.com/aarnphm/aarnphm.github.io/issues/56214) \[Model\] Support DeepSeek-V4.1-Flash

[vllm-project/vllm#56214](https://github.com/vllm-project/vllm/pull/56214) · zyongye · MERGED · created 2026-09-10T07:24:24Z · merged 2026-09-11T09:11:20Z

---

## comments

**mergify** (2026-09-10T07:26:35Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @zyongye.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**tjtanaa** (2026-09-10T09:14:10Z):

/amd-ci run

**github-actions** (2026-09-10T09:14:28Z):

✅ Triggered [Buildkite AMD CI #12816](https://buildkite.com/vllm/amd-ci/builds/12816) for commit `e47aa780bccf`.

**ChuanLi1101** (2026-09-11T00:51:23Z):

Thanks @zyongye — I went through quite a bit of the AMD path and this is really solid work. I especially appreciate that many of the changes are not just ports of the CUDA implementation, but are designed specifically for ROCm/gfx950.

It would be great to keep collaborating closely and avoid duplicating effort. My suggestion is that InferAct continues to drive the model/architecture-specific implementation, while AMD focuses on the platform side: AITER/kernel optimization, ROCm/runtime and CUDA graph issues, MI355X CI/recipe coverage, and E2E performance validation.

Markus has already identified the breakable CUDA graph issue and opened [#56349](https://github.com/aarnphm/aarnphm.github.io/issues/56349). The native segfault under `FULL_DECODE_ONLY` may be a separate runtime/correctness issue; we can help loop in the right AMD folks on that.

Thanks again for the great work. Please feel free to tag me on ROCm/MI355X-specific issues and we can coordinate follow-ups as they come up.

**zyongye** (2026-09-11T01:05:40Z):

> Thanks @zyongye — I went through quite a bit of the AMD path and this is really solid work. I especially appreciate that many of the changes are not just ports of the CUDA implementation, but are designed specifically for ROCm/gfx950.
>
> It would be great to keep collaborating closely and avoid duplicating effort. My suggestion is that InferAct continues to drive the model/architecture-specific implementation, while AMD focuses on the platform side: AITER/kernel optimization, ROCm/runtime and CUDA graph issues, MI355X CI/recipe coverage, and E2E performance validation.
>
> Markus has already identified the breakable CUDA graph issue and opened [#56349](https://github.com/aarnphm/aarnphm.github.io/issues/56349). The native segfault under `FULL_DECODE_ONLY` may be a separate runtime/correctness issue; we can help loop in the right AMD folks on that.
>
> Thanks again for the great work. Please feel free to tag me on ROCm/MI355X-specific issues and we can coordinate follow-ups as they come up.

@ChuanLi1101 Thank you!!! We are also very happy have AMD support on this release and would love to learn more from AMD side on kernel and runtime optimizations.

We will hopefully get the model merged ASAP and we can start looking into performance optimizations together.

I would also like to shout out to all my colleagues at Inferact, especially @jeejeelee who is leading the AMD effort on this model. This release wouldn’t be possible with all of them.

Let’s stay in touch, Cheers!!

**andyluo7** (2026-09-11T01:53:12Z):

Hi @zyongye — while validating [#56214](https://github.com/aarnphm/aarnphm.github.io/issues/56214) on 8x MI300X/gfx942, we found one gfx942-specific correctness issue.

On gfx942, `EmulationMxfp8LinearKernel.process_weights_after_loading()` can replace the original one-byte MXFP8 `wo_a.weight` with BF16 while retaining `weight_scale` or `weight_scale_inv` as metadata. `_get_cached_wo_a_bf16()` then interprets the retained scale as evidence that the weight is still FP8 and applies it again, causing double dequantization before the TP all-reduce.

The fix is to apply block scaling only while the weight still uses one-byte storage:



```python
if wo_a_scale_param is not None and wo_a.weight.element_size() == 1:
```

I also added regression coverage for already-dequantized BF16 weights retaining either scale attribute.

Validation on 8x MI300X/gfx942, TP8:

- Focused ROCm tests: `4 passed, 61 deselected`
- Direct retained-scale probes: exact BF16 match for both `weight_scale` and `weight_scale_inv`; the unfixed path produced an output/expected norm ratio of `0.25`
- Full official GSM8K evaluator: `1197/1319 = 90.7506%`
- GSM8K contract: 8-shot, 1,319 questions, max tokens 256, temperature 0, seed 42, no explicit concurrency override
- Zero invalid responses, zero API errors, and zero fatal server-log patterns
- Full model-default 1,048,576-token context, `FULL_AND_PIECEWISE` graphs, DSpark-5, and `AITER_TRITON_MXFP4_BF16`

The evaluator is byte-identical to the one used by [#56342](https://github.com/aarnphm/aarnphm.github.io/issues/56342), and the MI300X result is within that PR’s published fused MI355X range of 89.76%-91.36%.

I have this prepared as a signed one-commit patch. Since [#56214](https://github.com/aarnphm/aarnphm.github.io/issues/56214) currently needs a rebase against `main`, would you prefer to cherry-pick it, or should I push it directly to `dsv41-feat` after the rebase?

**jeejeelee** (2026-09-11T02:09:50Z):

@andyluo7 you can push it directly after we rebase

**andyluo7** (2026-09-11T02:46:45Z):

Thanks — I rebased the gfx942 fix onto the new [#56214](https://github.com/aarnphm/aarnphm.github.io/issues/56214) head (`48f1cd0da6`). The resulting signed commit is:

[andyluo7/vllm@`c4c90aa`](https://github.com/andyluo7/vllm/commit/c4c90aae20954f6821072284eb132fc3e31f3eee)

My direct push to `vllm-project/vllm:dsv41-feat` was rejected with HTTP 403, so I pushed the exact commit to my fork instead. You can apply it with:



```bash
git fetch https://github.com/andyluo7/vllm.git dsv41-gfx942-followup-20260910
git cherry-pick c4c90aae20954f6821072284eb132fc3e31f3eee
```

The rebased diff is still limited to the WO_A one-byte storage guard and its two retained-scale regression cases. Ruff, formatting, and `git diff --check` all pass. The test file is byte-identical to the hardware-tested version, and the affected helper is semantically identical apart from the variable rename introduced during the PR rebase.

**jeejeelee** (2026-09-11T02:53:55Z):

@andyluo7 done

**jeejeelee** (2026-09-11T02:56:51Z):

/ci run

**github-actions** (2026-09-11T02:57:07Z):

✅ Triggered [Buildkite CI #88234](https://buildkite.com/vllm/ci/builds/88234) for commit `172d9a171172`.

**ZJY0516** (2026-09-11T04:37:55Z):

/ci run

**github-actions** (2026-09-11T04:38:13Z):

✅ Triggered [Buildkite CI #88249](https://buildkite.com/vllm/ci/builds/88249) for commit `0bfb653d3b51`.

**ZJY0516** (2026-09-11T07:36:02Z):

/ci run

**github-actions** (2026-09-11T07:36:20Z):

✅ Triggered [Buildkite CI #88273](https://buildkite.com/vllm/ci/builds/88273) for commit `220d7027a6c1`.

**ZJY0516** (2026-09-11T07:57:21Z):

/ci run

**github-actions** (2026-09-11T07:57:39Z):

✅ Triggered [Buildkite CI #88279](https://buildkite.com/vllm/ci/builds/88279) for commit `7d81d62702b4`.

