---
title: '59443'
pageLayout: default
slug: thoughts/inference/vllm-prs/59443
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/59443.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#59443](https://github.com/aarnphm/aarnphm.github.io/issues/59443) \[Bugfix\]\[Qwen4Exp\] Load PLE tables unquantized under Quark checkpoints

[vllm-project/vllm#59443](https://github.com/vllm-project/vllm/pull/59443) · mjkvaak-amd · MERGED · created 2026-09-30T14:13:00Z · merged 2026-10-05T10:40:03Z

## Purpose

`amd/Qwen3.8-Flash-Next-Quark-MXFP4` no longer loads on `main`. Fail at model construction:



```text
NotImplementedError: Qwen4Exp PLE embedding does not support quantization config QuarkConfig
```

`Qwen4ExpPLEEmbeddingMethod.from_quant_config` handles `None`, FP8 and the ModelOpt configs, and raises for everything else. It came in with [#57497](https://github.com/aarnphm/aarnphm.github.io/issues/57497) (PLE n-gram table CPU offload). Before that, the Quark checkpoint served normally, for example on `nightly-cd10ed6f9`.

Quark quantizes only Linear and MoE layers, and this checkpoint’s PLE shards (`ple.ple_embedding.ngram_embedding.shard_*.weight`) are BF16 with no scales. So under a `QuarkConfig` the table is unquantized, and this PR returns `Qwen4ExpPLEUnquantizedEmbeddingMethod`.

`test_ple_embedding_is_unquantized_under_quark` covers the new branch. The existing rejection test, for NVFP4 and online FP8, is unchanged.

---

## Test Plan

- Hardware: MI355X (gfx950)
- Unit tests: `tests/models/qwen4_exp/test_ple.py -k "quant or exclusion or quark or inc"`
- Serving: `amd/Qwen3.8-Flash-Next-Quark-MXFP4`, TP2, the model-card recipe, and GSM8K 5-shot (chat template, thinking off), on the latest ROCm nightly `vllm/vllm-openai-rocm:nightly-18f8f96025` with only this PR applied

---

## Test Result

### Unit tests



```bash
python -m pytest -q tests/models/qwen4_exp/test_ple.py -k "quant or exclusion or quark or inc"
```



```text
7 passed, 69 deselected, 14 warnings in 5.35s
```

### Serving

On the unpatched nightly the server does not start (see Purpose). With this PR:



```bash
vllm serve amd/Qwen3.8-Flash-Next-Quark-MXFP4 --tensor-parallel-size 2 \
  --max-model-len 262144 --max-num-seqs 256 --gpu-memory-utilization 0.9 \
  --no-enable-prefix-caching --reasoning-parser qwen3 --trust-remote-code \
  --default-chat-template-kwargs '{"enable_thinking":false}'
# env: VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1

lm_eval --model local-chat-completions --tasks gsm8k --num_fewshot 5 \
  --apply_chat_template --batch_size 32 --seed 4321 \
  --model_args model=amd/Qwen3.8-Flash-Next-Quark-MXFP4,base_url=http://localhost:8000/v1/chat/completions,num_concurrent=32,max_gen_toks=2048,tokenized_requests=False
```

| Tasks | Version | Filter           | n-shot | Metric      |   |  Value |   | Stderr |
| ----- | ------: | ---------------- | -----: | ----------- | - | -----: | - | -----: |
| gsm8k |       3 | flexible-extract |      5 | exact_match | ↑ | 0.9629 | ± | 0.0052 |
|       |         | strict-match     |      5 | exact_match | ↑ | 0.9613 | ± | 0.0053 |

Quality matches with <https://huggingface.co/amd/Qwen3.8-Flash-Next-Quark-MXFP4#accuracy> . No accuracy regressions.

---

<details>
<summary> Essential Elements of an Effective PR Description Checklist </summary>

- [x] The purpose of the PR, such as “Fix some issue (link existing issues this PR will resolve)”.
- [x] The test plan, such as providing test command.
- [x] The test results, such as pasting the results comparison before and after, or e2e results
- [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.

</details>

---

Co-authored-by: Cursor <cursoragent@cursor.com>

---

## comments

**mergify** (2026-09-30T17:41:26Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @mjkvaak-amd.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**tjtanaa** (2026-10-02T06:41:54Z):

/ci run

**github-actions** (2026-10-02T06:42:05Z):

✅ @mjkvaak-amd, CI is now available for this PR.

- `/ci run` starts upstream CI; `/amd-ci run` starts AMD CI only.
- Your branch must contain every commit currently on its upstream target branch. Merge or rebase onto the latest target branch, then rerun the command. Append `--allow-stale` to a run command to test an outdated branch at your own risk.
- `/ci retry` retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
- `/amd-ci retry` retries failed jobs in AMD CI for the current PR head. Use `/amd-ci run` when the current head has no AMD CI build.
- `/ci cancel` cancels scheduled or running CI builds for this PR branch; `/amd-ci cancel` does the same for AMD CI only.

<!-- vllm-ci-authorized -->

**github-actions** (2026-10-02T06:42:13Z):

❌ This PR is 101 commits behind upstream `main`. Your branch must contain every commit currently on upstream `main`. No new CI build was started. Merge or rebase onto the latest `main`, then rerun `/ci run`. To test this branch at your own risk, use `/ci run --allow-stale`.

<!-- vllm-ci-command:5946866664 -->

**tjtanaa** (2026-10-02T06:42:46Z):

@mjkvaak-amd please provide end to end lm-eval score for this quark model with PLE enabled.

**mergify** (2026-10-05T02:06:22Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @mjkvaak-amd.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**mjkvaak-amd** (2026-10-05T05:27:13Z):

> @mjkvaak-amd please provide end to end lm-eval score for this quark model with PLE enabled.

I added the lm-eval score in the PR body.

**tjtanaa** (2026-10-05T09:29:11Z):

/ci run

**github-actions** (2026-10-05T09:29:31Z):

✅ Triggered [Buildkite CI #92861](https://buildkite.com/vllm/ci/builds/92861) for commit `db60ace563a7`.

**vllm-agent** (2026-10-05T09:54:44Z):

<!-- ci-selector-shadow -->

### CI selector (shadow): 1 test steps (1 jobs) instead of 36 (51 jobs)

Shadow mode: this changes nothing about what CI runs. It shows what the evidence-based selector would pick for this PR, next to today’s rules. [How it works](https://github.com/vllm-project/ci-infra/tree/main/buildkite/ci_selector).

**Feedback welcome:** reply here if it would skip a step this change needs, or runs something unrelated.

| steps (jobs)           | Today’s rules | Selector | Would skip | Would add |
| ---------------------- | ------------- | -------- | ---------- | --------- |
| NVIDIA, CPU and others | 36 (51)       | 1 (1)    | 35 (50)    | 0 (0)     |
| AMD mirrors            | 33 (43)       | 1 (1)    | 32 (42)    | 0 (0)     |

<details><summary>Selector would run (1)</summary>

- `qwen4-exp-unit-tests-cpu`

</details>

<details><summary>Would skip (today's rules run them) (35)</summary>

- `ascend-npu-test`
- `basic-correctness` ×2
- `basic-correctness-cpu-offload`
- `basic-correctness-cumem`
- `basic-correctness-prefetch-offload`
- `basic-correctness-sleep-mode`
- `basic-models-test-other-cpu`
- `basic-models-tests-initialization`
- `basic-models-tests-other`
- `benchmarks-cli-test`
- `cpu-language-generation-and-pooling-model-tests` ×3
- `distributed-model-tests-2-gpus` ×3
- `entrypoints-integration-api-server` ×4
- `entrypoints-integration-api-server-generate`
- `entrypoints-integration-api-server-openai-chat_completion`
- `entrypoints-integration-api-server-openai-completion`
- `entrypoints-integration-llm`
- `entrypoints-integration-multimodal`
- `entrypoints-integration-pooling`
- `entrypoints-integration-responses-api`
- `entrypoints-integration-speech_to_text`
- `kernels-fla-ops-test-b200`
- `kernels-mhc-test-b200`
- `kernels-root-misc-test-b200`
- `language-models-tests-granite-l4-compatibility`
- `language-models-tests-hybrid` ×2
- `language-models-tests-standard`
- `multi-modal-models-standard-1-qwen2`
- `multi-modal-models-standard-2-qwen3-gemma`
- `multi-modal-models-standard-3-llava-qwen2-vl`
- `multi-modal-models-standard-4-other-whisper`
- `multi-modal-processor` ×4
- `multi-modal-processor-cpu` ×4
- `qwen4-exp-unit-tests`
- `v1-spec-decode`

</details>

<details><summary>Would add (today's rules do not run them) (0)</summary>

none

</details>

<details><summary>AMD mirrors: would skip (32)</summary>

- `basic-correctness` ×2
- `basic-correctness-cpu-offload`
- `basic-correctness-cumem`
- `basic-correctness-prefetch-offload`
- `basic-correctness-sleep-mode`
- `basic-models-tests-initialization`
- `basic-models-tests-other`
- `benchmarks-cli-test`
- `distributed-model-tests-2-gpus` ×3
- `entrypoints-integration-api-server` ×4
- `entrypoints-integration-api-server-generate`
- `entrypoints-integration-api-server-openai-chat_completion`
- `entrypoints-integration-api-server-openai-completion`
- `entrypoints-integration-llm`
- `entrypoints-integration-multimodal`
- `entrypoints-integration-pooling`
- `entrypoints-integration-responses-api`
- `entrypoints-integration-speech_to_text`
- `kernels-fla-ops-test-b200`
- `kernels-mhc-test-b200`
- `kernels-root-misc-test-b200`
- `language-models-tests-granite-l4-compatibility`
- `language-models-tests-hybrid` ×2
- `language-models-tests-standard`
- `multi-modal-models-standard-1-qwen2`
- `multi-modal-models-standard-2-qwen3-gemma`
- `multi-modal-models-standard-3-llava-qwen2-vl`
- `multi-modal-models-standard-4-other-whisper`
- `multi-modal-processor` ×4
- `platform-tests`
- `pytorch-compilation-passes-unit-tests`
- `v1-spec-decode`

</details>

<details><summary>AMD mirrors: would add (0)</summary>

none

</details>

<sub>2 changed files · base `b1f229fb75` · head `db60ace563` · Python record: build 92817 at `7867d6c52d` · kernel record: table [`7867d6c`](https://github.com/aarnphm/aarnphm.github.io/commit/7867d6c52d) (build 92817), map [`7867d6c`](https://github.com/aarnphm/aarnphm.github.io/commit/7867d6c52d) · not counted: 10 build steps, 5 A100 steps the generator no longer emits, 3 optional steps the selector would also run</sub>

