---
title: '60021'
pageLayout: default
slug: thoughts/inference/vllm-prs/60021
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/60021.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#60021](https://github.com/aarnphm/aarnphm.github.io/issues/60021) \[ROCm\]\[Qwen4Exp\] Use fused PLE Triton kernels on the AMD backend

[vllm-project/vllm#60021](https://github.com/vllm-project/vllm/pull/60021) · mjkvaak-amd · MERGED · created 2026-10-05T07:19:21Z · merged 2026-10-08T15:47:57Z

## Overview

Ports the fused Qwen4Exp PLE Triton kernels from [#54517](https://github.com/vllm-project/vllm/pull/54517) (with the [#55375](https://github.com/vllm-project/vllm/pull/55375) stride fix) to the AMD backend, which still runs the older PyTorch PLE path. Follows up on [#54764](https://github.com/vllm-project/vllm/issues/54764); supersedes [#58325](https://github.com/vllm-project/vllm/pull/58325).

## Claims

- Output throughput on Qwen3.8-Flash-Next MXFP4, TP2 on MI355X, rises by +1.2% to +2.6% at each serving point (geomean +1.7%, +2.2% and +1.6% over three repeats). GSM8K is unchanged within noise.
- Per decode step, the PLE layer drops from 19 kernel launches to 3.
- The AMD backend no longer builds padded `[num_reqs, max_len]` n-gram buffers.
- NVIDIA and AMD now share one PLE kernel module. Net −360 non-test lines (+226 / −586).

## Validation

**Unit tests.** MI355X (gfx950), `vllm/vllm-openai-rocm:nightly-18f8f96025` with this branch’s `vllm/models/qwen4_exp` and `tests/models/qwen4_exp` mounted (the nightly and this branch’s base have no other differences in that package):



```text
pytest tests/models/qwen4_exp/test_ple.py
90 passed, 15 warnings in 27.71s
```

- `test_fused_conv_correctness` now also runs on `amd.ple_layer.Qwen4ExpPLELayer`, covering every decode, prefill, spec and mixed case and both conv-state layouts.
- New `test_amd_ngram_embedding_matches_reference_under_compile` checks the AMD n-gram embedding under `torch.compile(fullgraph=True, dynamic=True)` against the Python reference, including an empty request and EOS in the context.

**End to end.** `amd/Qwen3.8-Flash-Next-Quark-MXFP4`, TP2 on MI355X. Control and candidate are the same ROCm image, with and without this change. Mean of 3 repeats, control → this change:

| ISL / OSL | Conc | Output tok/s              | Mean TPOT (ms) | Per repeat          |
| --------- | ---- | ------------------------- | -------------- | ------------------- |
| 8k / 1k   | 4    | 537.5 → 551.4 (+2.6%)     | 6.82 → 6.64    | +2.1%, +3.0%, +2.7% |
| 8k / 1k   | 16   | 1,277.3 → 1,301.1 (+1.9%) | 11.35 → 11.16  | +1.6%, +2.5%, +1.4% |
| 8k / 1k   | 64   | 2,296.3 → 2,336.0 (+1.7%) | 24.48 → 24.04  | +1.6%, +2.1%, +1.6% |
| 60k / 600 | 4    | 231.1 → 233.7 (+1.2%)     | 10.49 → 10.39  | +1.7%, +1.0%, +0.8% |

GSM8K (5-shot, chat template, 1,319 questions, flexible-extract): 0.9639 → 0.9624 (mean of 3). **No regressions.**

| Repeat | Control → this change | Fixed / broken | Exact McNemar p |
| ------ | --------------------- | -------------- | --------------- |
| 1      | 0.9659 → 0.9598       | 11 / 19        | 0.20            |
| 2      | 0.9629 → 0.9629       | 18 / 18        | 1.00            |
| 3      | 0.9629 → 0.9644       | 18 / 16        | 0.86            |

**Decode trace.** Torch profiler on rank 0, 8k/128 at c16, 128 decode steps per arm:

|                              | Control  | This change                                    |
| ---------------------------- | -------- | ---------------------------------------------- |
| PLE kernel launches per step | 19       | 3 (`_ple_ngram_ids`, `_ple_gate`, `_ple_conv`) |
| PLE GPU time per step        | 134 µs   | 18 µs                                          |
| Decode step                  | 10.45 ms | 10.22 ms (−2.2%)                               |

**Reproduce:**



```bash
vllm serve amd/Qwen3.8-Flash-Next-Quark-MXFP4 --tensor-parallel-size 2 \
  --max-model-len 262144 --max-num-seqs 256 --gpu-memory-utilization 0.9 \
  --no-enable-prefix-caching --trust-remote-code --reasoning-parser qwen3 \
  --default-chat-template-kwargs '{"enable_thinking":false}' --port 8000

# per point: (ISL, OSL, conc, prompts) = (8192,1024,4,32) (8192,1024,16,64) (8192,1024,64,128) (60000,600,4,16)
python -m sglang.bench_serving --backend vllm --port 8000 --dataset-name random \
  --random-input-len 8192 --random-output-len 1024 --random-range-ratio 1.0 \
  --max-concurrency 16 --num-prompts 64 --warmup-requests 16 --request-rate inf

lm_eval --model local-chat-completions --tasks gsm8k --num_fewshot 5 --apply_chat_template \
  --model_args model=amd/Qwen3.8-Flash-Next-Quark-MXFP4,base_url=http://localhost:8000/v1/chat/completions,num_concurrent=32,max_gen_toks=2048 \
  --seed 4321 --log_samples --output_path gsm8k-out
```

## Details

**Why the kernels port unchanged.** They are plain Triton. Their PDL path is gated on `current_platform.is_arch_support_pdl()`, which is false on ROCm, so `nvidia/ops/ple.py` moves to `common/ops/ple.py` as is, with the NVIDIA imports updated.

**AMD model changes:**

- **`Qwen4ExpNGramEmbedding`:** calls `ple_ngram_ids` instead of building padded buffers with gathers, a shift and a scatter, and drops the `max_num_reqs` argument. The pinned-host and device embedding lookups that follow are unchanged.
- **`Qwen4ExpPLELayer`:**
  - `key_proj` and `value_proj` become one replicated `MergedColumnParallelLinear` named `kv_proj` (`disable_tp=True`, as on NVIDIA).
  - The gate is `ple_gate`.
  - The short conv is `ple_conv`, which replaces the separate decode, prefill and spec `F.conv1d` branches with NVIDIA’s single dispatch.
  - `qwen4_exp_ple_short_conv` stays a splitting op.
- **Residual:** the fused conv also adds the decoder layer’s residual, so `Qwen4ExpDecoderLayer` now assigns `hidden_states = self.ple(...)` instead of adding the PLE output itself.
- **Weight loading:**
  - `ple.key_proj` and `ple.value_proj` map onto the two `kv_proj` shards.
  - `_HC_WEIGHTS_MAPPER` becomes `_EXTRA_WEIGHTS_MAPPER`, since it now also maps PLE weights.
  - `kv_proj` is added to `packed_modules_mapping`, so a quantization exclude list that names the original projections still applies. The Quark MXFP4 checkpoint keeps both in BF16.
  - The AMD MTP draft runs without PLE, so it only picks up the mapper rename.

**Design choice: a custom op around `ple_ngram_ids` on AMD.** Unlike the NVIDIA path, the AMD model runs the n-gram embedding inside `torch.compile`, where `query_start_loc` has a dynamic leading dimension. The kernel’s launch grid comes from the request count, which Dynamo would otherwise trace symbolically. So the call goes through a small opaque op, `qwen4_exp_amd_ple_ngram_ids`. The new compile test covers that path.

> Validated on MI355X (gfx950) only, not MI300X/MI325X (gfx942). The kernels contain no architecture-specific code.

---

<details>
<summary> Pull Request Checklist </summary>

- [x] I used vLLM’s `/pr-checklist` skill. (Mandatory for agents, optional for humans).

- [x] AI assistance was used during the creation of this PR.

- [x] **Design Fit:** Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

- [x] **Testing and Validation:** Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

- [x] **Code Quality and Style:** Keeps code and comments clear and concise, and updates relevant documentation and examples.

- [x] **Pull Request Contents:** Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

</details>

Co-authored-by: Cursor <cursoragent@cursor.com>

---

## comments

**gau-nernst** (2026-10-05T12:39:34Z):

For the record, since the main benefit of fused PLE kernels is to reduce memory usage (there’s only 1 PLE layer per forward pass, so any further speedup won’t matter much e2e), I think having the Triton kernels under `common` is fine, since we don’t expect much more changes to the kernels.

**mjkvaak-amd** (2026-10-07T11:55:46Z):

> Nice work!
>
> **Suggestion**: can we check that MTP still works as well? like a gsm8k
>
> **Optional**: we could also do a quick run on gfx942 to validate it also works as expected
>
> **Optional follow-up**: I can see kernels like \_ple_ngram_ids have launch parameters that are tuned for nvidia specifically (specifically, that one set num_warps=4). A follow-up item could be to tune these and make them platform aware in case we want to squeeze even more perf out of them

**MTP still works with the PR, and GSM8K doesn’t change.**

Setup: `amd/Qwen3.8-Flash-Next-Quark-MXFP4`, TP2 on MI355X.

|                            | Without this PR | With this PR |
| -------------------------- | --------------: | -----------: |
| GSM8K, flexible-extract    |          0.9659 |       0.9644 |
| GSM8K, strict-match        |          0.9659 |       0.9636 |
| MTP mean acceptance length |            3.52 |         3.51 |

🚨One thing to know before you try yourself: on the current nightly, MTP fails to load this checkpoint even without this PR. Every MTP tensor is BF16, but the checkpoint’s Quark `exclude` list names the MTP modules `mtp.layers.0.*`, while vLLM builds the draft as `mtp.layers.48.*`. The AMD MTP code renames `ignored_layers` / `exclude_modules` to fix this up, but `QuarkConfig` keeps its list in `quant_config["exclude"]`, so nothing gets renamed. vLLM then builds the draft’s routed experts as MXFP4, and loading fails with `size of tensor a (1280) must match the size of tensor b (2560)`. For both arms above I added `re:mtp\..*` to the exclude list in `config.json` and changed nothing else. Fix is out of the scope of this PR.

---

**gfx942 performance is consistent with the change**

Setup: `Qwen/Qwen3.8-Flash-Next-FP8`, TP2 on MI300X.

> Notice the different checkpoint, since MXFP4 won’t run on gfx94

Unit tests pass, the server comes up on the fused path, throughput is +0.8% geomean, and GSM8K is unchanged.

| ISL / OSL | Conc | Output tok/s          | Mean TPOT (ms) | Per repeat   |
| --------- | ---- | --------------------- | -------------- | ------------ |
| 8k / 1k   | 4    | 281.8 → 284.1 (+0.8%) | 12.44 → 12.30  | +0.9%, +0.8% |
| 8k / 1k   | 16   | 562.3 → 567.9 (+1.0%) | 23.75 → 23.51  | +1.2%, +0.8% |
| 8k / 1k   | 64   | 825.7 → 831.9 (+0.8%) | 62.55 → 62.07  | +0.7%, +0.8% |
| 60k / 600 | 4    | 81.0 → 81.5 (+0.7%)   | 27.13 → 26.88  | +0.8%, +0.6% |

That is smaller than on MI355X, as expected: the PLE layer saves the same few launches, but each decode step is about twice as long on MI300X.

GSM8K (5-shot, chat template, flexible-extract, 1,319 questions): 0.9712 → 0.9727, 4 fixed / 2 broken, exact McNemar p = 0.69.

🚨Two issues outside this PR, both present on the unmodified nightly-43b4aaea3e40e20ef53ce2bfecfa1b72060bf2b5 with this FP8 checkpoint on gfx942:

- With `VLLM_ROCM_USE_AITER=1`, AITER’s FP8 MoE is selected and raises instead of falling back. At TP2 the 128x128 block scales are refined to 64x64, which it does not support.
- `common/ngram_embedding.py` all-reduces the FP8 n-gram rows as `int8`. AITER’s custom all-reduce accepts the tensor in `should_custom_ar`, then its kernel rejects it (“custom allreduce only supports float32, float16 and bfloat16”). vLLM’s own custom all-reduce checks the dtype and falls back to RCCL.

Fixing either is out of the scope of this PR.

---

**AMD specific tuning**

IMO tuning these kernels isn’t worth it on AMD. Even if the launch parameters were tuned perfectly, decode would get about 0.1% faster, which is too small to measure in serving.

| Kernel          | Median per call | Launch parameters                  |
| --------------- | --------------- | ---------------------------------- |
| \_ple_ngram_ids | 4.6 µs          | BLOCK_T=8, num_warps=4             |
| \_ple_gate      | 4.8 µs          | num_warps=4                        |
| \_ple_conv      | 8.6 µs          | BLOCK_C=512, num_warps=8 in decode |
| Total           | 18 µs           | 0.18% of the step                  |

**github-actions** (2026-10-07T14:37:15Z):

✅ @mjkvaak-amd, CI is now available for this PR.

- `/ci run` starts upstream CI; `/amd-ci run` starts AMD CI only.
- Your branch must contain every commit currently on its upstream target branch. Merge or rebase onto the latest target branch, then rerun the command. Append `--allow-stale` to a run command to test an outdated branch at your own risk.
- `/ci retry` retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
- `/amd-ci retry` retries failed jobs in AMD CI for the current PR head. Use `/amd-ci run` when the current head has no AMD CI build.
- `/ci cancel` cancels scheduled or running CI builds for this PR branch; `/amd-ci cancel` does the same for AMD CI only.

<!-- vllm-ci-authorized -->

**dllehr-amd** (2026-10-07T14:37:16Z):

/ci run

**github-actions** (2026-10-07T14:37:41Z):

✅ Triggered [Buildkite CI #93334](https://buildkite.com/vllm/ci/builds/93334) for commit `21d533b537c2`.

**vllm-agent** (2026-10-07T14:58:41Z):

<!-- ci-selector-shadow -->

### CI selector (shadow): 5 test steps (5 jobs) instead of 37 (52 jobs)

Shadow mode: this changes nothing about what CI runs. It shows what the evidence-based selector would pick for this PR, next to today’s rules. [How it works](https://github.com/vllm-project/ci-infra/tree/main/buildkite/ci_selector).

**Feedback welcome:** reply here if it would skip a step this change needs, or runs something unrelated.

| steps (jobs)           | Today’s rules | Selector | Would skip | Would add |
| ---------------------- | ------------- | -------- | ---------- | --------- |
| NVIDIA, CPU and others | 37 (52)       | 5 (5)    | 35 (50)    | 3 (3)     |
| AMD mirrors            | 34 (44)       | 8 (8)    | 31 (41)    | 5 (5)     |

<details><summary>Selector would run (5)</summary>

- `cpu-multimodal-config`
- `distributed-dp-tests-4-gpus`
- `engine`
- `qwen4-exp-unit-tests`
- `qwen4-exp-unit-tests-cpu`

</details>

<details><summary>Would skip (today's rules run them) (35)</summary>

- `ascend-npu-test`
- `basic-correctness` ×2
- `basic-correctness-cpu-offload`
- `basic-correctness-cumem`
- `basic-correctness-prefetch-offload`
- `basic-correctness-sleep-mode`
- `basic-models-test-other-cpu`
- `basic-models-tests-initialization`
- `basic-models-tests-other`
- `benchmarks-cli-test`
- `cpu-language-generation-and-pooling-model-tests` ×3
- `distributed-model-tests-2-gpus` ×3
- `entrypoints-integration-api-server` ×4
- `entrypoints-integration-api-server-generate`
- `entrypoints-integration-api-server-openai-chat_completion`
- `entrypoints-integration-api-server-openai-completion`
- `entrypoints-integration-llm`
- `entrypoints-integration-multimodal`
- `entrypoints-integration-pooling`
- `entrypoints-integration-responses-api`
- `entrypoints-integration-speech_to_text`
- `kernels-fla-ops-test-b200`
- `kernels-mhc-test-b200`
- `kernels-root-misc-test-b200`
- `language-models-tests-granite-l4-compatibility`
- `language-models-tests-hybrid` ×2
- `language-models-tests-standard`
- `multi-modal-models-standard-1-qwen2`
- `multi-modal-models-standard-2-qwen3-gemma`
- `multi-modal-models-standard-3-llava-qwen2-vl`
- `multi-modal-models-standard-4-other-whisper`
- `multi-modal-processor` ×4
- `multi-modal-processor-cpu` ×4
- `rl-entrypoints-tests`
- `v1-spec-decode`

</details>

<details><summary>Would add (today's rules do not run them) (3)</summary>

- `cpu-multimodal-config` (code map)
- `distributed-dp-tests-4-gpus` (code map)
- `engine` (code map)

</details>

<details><summary>AMD mirrors: would skip (31)</summary>

- `basic-correctness` ×2
- `basic-correctness-cpu-offload`
- `basic-correctness-cumem`
- `basic-correctness-prefetch-offload`
- `basic-correctness-sleep-mode`
- `basic-models-tests-initialization`
- `benchmarks-cli-test`
- `distributed-model-tests-2-gpus` ×3
- `entrypoints-integration-api-server` ×4
- `entrypoints-integration-api-server-generate`
- `entrypoints-integration-api-server-openai-chat_completion`
- `entrypoints-integration-api-server-openai-completion`
- `entrypoints-integration-llm`
- `entrypoints-integration-multimodal`
- `entrypoints-integration-pooling`
- `entrypoints-integration-responses-api`
- `entrypoints-integration-speech_to_text`
- `kernels-fla-ops-test-b200`
- `kernels-mhc-test-b200`
- `kernels-root-misc-test-b200`
- `language-models-tests-granite-l4-compatibility`
- `language-models-tests-hybrid` ×2
- `language-models-tests-standard`
- `multi-modal-models-standard-1-qwen2`
- `multi-modal-models-standard-2-qwen3-gemma`
- `multi-modal-models-standard-3-llava-qwen2-vl`
- `multi-modal-models-standard-4-other-whisper`
- `multi-modal-processor` ×4
- `platform-tests`
- `pytorch-compilation-passes-unit-tests`
- `v1-spec-decode`

</details>

<details><summary>AMD mirrors: would add (5)</summary>

- `batch-invariance-h100` (code map)
- `engine` (code map)
- `pytorch-nightly-dependency-override-check` (code map)
- `ray-dependency-compatibility-check` (code map)
- `torch-stable-abi-audit` (code map)

</details>

<sub>9 changed files · base `a58bdd0d4e` · head `64bb4e8866` · Python record: build 93039 at `1e5d0ea888` · kernel record: table [`43b4aae`](https://github.com/aarnphm/aarnphm.github.io/commit/43b4aaea3e) (build 93244), map [`43b4aae`](https://github.com/aarnphm/aarnphm.github.io/commit/43b4aaea3e) · not counted: 10 build steps, 5 A100 steps the generator no longer emits, 1 optional steps the selector would also run</sub>

**simondanielsson** (2026-10-07T15:59:18Z):

Thanks for following up. the gfx942 and tuning parts look good.

> MTP still works with the PR, and GSM8K doesn’t change.
> Setup: amd/Qwen3.8-Flash-Next-Quark-MXFP4, TP2 on MI355X.
> 🚨One thing to know before you try yourself: on the current nightly, MTP fails to load this checkpoint even without this PR. Every MTP tensor is BF16, but the checkpoint’s Quark exclude list names the MTP modules mtp.layers.0._, while vLLM builds the draft as mtp.layers.48._.

If MTP doesn’t work for mxfp4, can it instead be verified on the non-quantized model?

**mjkvaak-amd** (2026-10-07T19:05:55Z):

> Thanks for following up. the gfx942 and tuning parts look good.
>
> > MTP still works with the PR, and GSM8K doesn’t change.
> > Setup: amd/Qwen3.8-Flash-Next-Quark-MXFP4, TP2 on MI355X.
> > 🚨One thing to know before you try yourself: on the current nightly, MTP fails to load this checkpoint even without this PR. Every MTP tensor is BF16, but the checkpoint’s Quark exclude list names the MTP modules mtp.layers.0._, while vLLM builds the draft as mtp.layers.48._.
>
> If MTP doesn’t work for mxfp4, can it instead be verified on the non-quantized model?

MTP works on the FP8 checkpoint without hacks, and stock and the PR agree. The unmodified checkpoint loaded the MTP draft out of the box in both arms, and there were no errors in either server log.

|                            | Stock  | With this PR |
| -------------------------- | ------ | ------------ |
| GSM8K, flexible-extract    | 0.9704 | 0.9704       |
| GSM8K, strict-match        | 0.9712 | 0.9712       |
| MTP mean acceptance length | 3.51   | 3.52         |

**Setup.** `Qwen/Qwen3.8-Flash-Next-FP8`, TP2 on MI300X (gfx942), `--speculative-config '{"method":"mtp","num_speculative_tokens":3}'`. Stock is the unmodified `nightly-43b4aaea3e`; the candidate is the same image with this PR’s diff applied.

**mjkvaak-amd** (2026-10-07T19:07:12Z):

/ci run

**github-actions** (2026-10-07T19:07:36Z):

✅ Triggered [Buildkite CI #93387](https://buildkite.com/vllm/ci/builds/93387) for commit `7d61c94424e5`.

**gau-nernst** (2026-10-08T00:14:56Z):

/ci retry

**github-actions** (2026-10-08T00:15:16Z):

✅ Triggered [Buildkite CI #93463](https://buildkite.com/vllm/ci/builds/93463) for commit `64bb4e88667f`, running 2 failed step(s) from [Buildkite CI #93387](https://buildkite.com/vllm/ci/builds/93387).

**tjtanaa** (2026-10-08T13:24:19Z):

@mjkvaak-amd Is Qwen4exp still running with model runner v1? I noticed that you are registering the ops with `direct_register_customop`. NVIDIA implementation doesn’t do that because they do not rely on torch compile. Model runner V1 is going to be deprecated soon where we don’t rely on torch compile. Will you start migrating to model runner v2 and close the gap between model runner V1?

**mjkvaak-amd** (2026-10-08T13:44:46Z):

> @mjkvaak-amd Is Qwen4exp still running with model runner v1? I noticed that you are registering the ops with `direct_register_customop`. NVIDIA implementation doesn’t do that because they do not rely on torch compile. Model runner V1 is going to be deprecated soon where we don’t rely on torch compile. Will you start migrating to model runner v2 and close the gap between model runner V1?

Qwen4Exp already runs on Model Runner V2, on both backends. Its PLE inputs (`query_start_loc`, `ngram_context`) come from `Qwen4ExpModelState`, a `MambaHybridModelState` from `vllm/v1/worker/gpu/model_states/`, and nothing in the V1 runner builds them. The architecture is not in `ROCM_DEFAULT_MRV1_ARCHITECTURES`, so ROCm takes V2 by default too. This PR does not depend on V1.

The registered ops come from how each backend is graphed, not from the runner. V2 still runs `torch.compile` and piecewise graphs. The AMD `Qwen4ExpModel` and its MTP draft use `@support_torch_compile`.

**tjtanaa** (2026-10-08T14:30:29Z):

/ci run

**github-actions** (2026-10-08T14:32:02Z):

✅ Triggered [Buildkite CI #93609](https://buildkite.com/vllm/ci/builds/93609) for commit `fd7f930a7681`.

