---
title: '59990'
pageLayout: default
slug: thoughts/inference/vllm-prs/59990
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/59990.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#59990](https://github.com/aarnphm/aarnphm.github.io/issues/59990) \[Bugfix\] Fix Qwen4Exp PLE embedding rejecting INC (AutoRound) checkpoints

[vllm-project/vllm#59990](https://github.com/vllm-project/vllm/pull/59990) · vadiklyutiy · MERGED · created 2026-10-05T00:06:48Z · merged 2026-10-05T01:52:08Z

## Overview

Fixes [#59798](https://github.com/aarnphm/aarnphm.github.io/issues/59798). `Intel/Qwen3.8-Flash-Next-W4A16-AutoRound` is an AutoRound checkpoint, which vLLM loads as `INCConfig`. It failed at model construction with `NotImplementedError: Qwen4Exp PLE embedding does not support quantization config INCConfig`. `Qwen4ExpPLEEmbeddingMethod.from_quant_config` now asks `INCConfig` whether the PLE layer is quantized. [#59431](https://github.com/aarnphm/aarnphm.github.io/issues/59431) does the same for compressed-tensors.

## Claims

- The command from the issue (TP=2, expert parallel) no longer raises. Both workers build the PLE n-gram table with `Qwen4ExpPLEUnquantizedEmbeddingMethod` (bf16), and model construction finishes.
- The checkpoint’s own INC metadata makes the choice: its `extra_config` has `".*ple.*": {"bits": 16, "data_type": "float"}`. If INC says the PLE layer is quantized, the old `NotImplementedError` is still raised, because there is no INC-quantized PLE format.
- The branch is in `vllm/models/qwen4_exp/common/ngram_embedding.py`, so both the NVIDIA and the AMD Qwen4Exp paths get it.
- A new unit test covers both cases.

## Validation

Hardware: 2 × NVIDIA A100-SXM4-80GB. Before: commit `18f8f96025b556071eb627076f94df560fbd3a22`. After: this PR on top of that commit.

Reproducer, from the issue. No request is needed, because the crash happens at startup:



```bash
VLLM_ENGINE_READY_TIMEOUT_S=3600 vllm serve Intel/Qwen3.8-Flash-Next-W4A16-AutoRound \
  --tensor-parallel-size 2 --enable-expert-parallel
```

\|        | Runs | Result                                                                                                                                                                                                                                                                        |
\| ------ | ---- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --- | ----- | --- | ----------------------------------------------------------------------------------------------------------------------- |
\| Before | 1    | Both workers raise `NotImplementedError: Qwen4Exp PLE embedding does not support quantization config INCConfig` during model construction, before weight loading. The engine exits with code 1. The `vllm/vllm-openai:nightly` image at the same commit gives the same error. |     | After | 2   | No `NotImplementedError` in either run. Both workers build the PLE table, and model construction finishes (about 56 s). |

After the fix, both workers log:



```
Initialized PLE embedding language_model.model.layers.1.ple.ple_embedding.ngram_embedding: quantization_method=Qwen4ExpPLEUnquantizedEmbeddingMethod, weight_dtype=torch.bfloat16, weight_device=cpu, pinned=True
```

Limitation: neither “after” run reached a ready server. Both runs next stopped at the weight download, with a `PermissionError` from `huggingface_hub`. The HF cache on the test machine had only the config and tokenizer files for this model, and we could not write to it. This happens after the bug site, in code the patch does not touch. The “before” run never got that far. No run here loaded the real weights, and no accuracy eval was run. Before merge, someone with the weights should run the reproducer to a ready server and run an accuracy check.

Extra checks (not the reproducer):

- The same command with `--load-format dummy`: weights loaded (35.44 GiB per GPU), CUDA graphs were captured, and the server became ready. One `/v1/completions` request returned 16 tokens. The text is noise, because the weights are random.
- The checkpoint’s safetensors headers on the Hub: the PLE n-gram table is 128 plain BF16 shards (`[2500012, 160]` each), with no `qweight`, `scales` or `qzeros`. This matches the unquantized method.
- Unit tests: `pytest --noconftest tests/models/qwen4_exp/test_ple.py -k "inc_layer_config or rejects_unsupported or modelopt_exclusion or shard_exclusions or mixed_precision"` gives `6 passed, 69 deselected`. `--noconftest` was used because `tests/conftest.py` needs `tblib`, which the test env did not have. With the INC branch turned off in a separate process (tree unchanged), the new test’s config gets the old `NotImplementedError` again. So the test catches the bug.
- `pre-commit run --files` on the two changed files: every hook passed or was skipped, including ruff and mypy.

## Details

Root cause: `from_quant_config` gets the model-level quantization config. That config describes the model body, which is not always the same as the PLE table. The method could only ask four config families whether the PLE layer is excluded: ModelOpt mixed precision, ModelOpt, compressed-tensors and FP8. Any other config fell through to the `NotImplementedError`. This checkpoint quantizes `model.language_model.layers` to 4 bits, but its `extra_config` marks `.*ple.*` as 16-bit float. `INCConfig` keeps this metadata and resolves it with `config_parser.resolve(layer, prefix)`, but the PLE code never asked it.

How it was found: on the unpatched tree, a debug print just before the raise showed three things. The config was a plain `INCConfig`, with no handled class in its MRO. The prefix was `language_model.model.layers.1.ple.ple_embedding.ngram_embedding`. And `resolve(None, prefix)` returned `bits=16, quantized=False`. So the INC metadata is already mapped to vLLM names and matches the prefix. Only the dispatch ignored it.

Fix: one `if` in `from_quant_config`. For an `INCConfig`, when `quant_config.config_parser.resolve(None, prefix).quantized` is false, it returns `Qwen4ExpPLEUnquantizedEmbeddingMethod`. `INCConfig.get_quant_method` uses the same `resolve(...).quantized` check for its own layers. Passing `None` as the layer is safe: `_resolve_raw` only uses the layer in `isinstance(layer, ParallelLMHead)` and `layer.__class__.__name__`, and both work with `None`. The compressed-tensors branch also passes `None`. Size: +6 lines in `ngram_embedding.py`, and +19 lines of test in `tests/models/qwen4_exp/test_ple.py`. That test file already runs in Buildkite (`.buildkite/test_areas/models_basic.yaml`).

Design choice for review: INC’s own `get_quant_method` treats every layer that is not linear or MoE as unquantized, whatever `resolve` says. This patch is stricter. An INC checkpoint that puts the PLE table inside a quantized block, with no 16-bit exclusion, still gets `NotImplementedError`. The new test checks this case on purpose. Please say if you prefer INC’s looser behavior.

---

<details>
<summary> Pull Request Checklist </summary>

- [x] I used vLLM’s `/pr-checklist` skill. (Mandatory for agents, optional for humans).

- [x] AI assistance was used during the creation of this PR.

- [x] **Design Fit:** Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

- [x] **Testing and Validation:** Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

- [x] **Code Quality and Style:** Keeps code and comments clear and concise, and updates relevant documentation and examples.

- [x] **Pull Request Contents:** Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide. </details>

---

## comments

**vadiklyutiy** (2026-10-05T00:39:18Z):

/ci run

**github-actions** (2026-10-05T00:39:37Z):

❌ This PR is 12 commits behind upstream `main`. Your branch must contain every commit currently on upstream `main`. No new CI build was started. Merge or rebase onto the latest `main`, then rerun `/ci run`. To test this branch at your own risk, use `/ci run --allow-stale`.

<!-- vllm-ci-command:5986152433 -->

**vadiklyutiy** (2026-10-05T00:40:15Z):

/ci run

**github-actions** (2026-10-05T00:41:57Z):

✅ Triggered [Buildkite CI #92821](https://buildkite.com/vllm/ci/builds/92821) for commit `9250458acf70`.

**vllm-agent** (2026-10-05T00:58:07Z):

<!-- ci-selector-shadow -->

### CI selector (shadow): 1 test steps (1 jobs) instead of 36 (51 jobs)

Shadow mode: this changes nothing about what CI runs. It shows what the evidence-based selector would pick for this PR, next to today’s rules. [How it works](https://github.com/vllm-project/ci-infra/tree/main/buildkite/ci_selector).

**Feedback welcome:** reply here if it would skip a step this change needs, or runs something unrelated.

| steps (jobs)           | Today’s rules | Selector | Would skip | Would add |
| ---------------------- | ------------- | -------- | ---------- | --------- |
| NVIDIA, CPU and others | 36 (51)       | 1 (1)    | 35 (50)    | 0 (0)     |
| AMD mirrors            | 33 (43)       | 1 (1)    | 32 (42)    | 0 (0)     |

<details><summary>Selector would run (1)</summary>

- `qwen4-exp-unit-tests-cpu`

</details>

<details><summary>Would skip (today's rules run them) (35)</summary>

- `ascend-npu-test`
- `basic-correctness` ×2
- `basic-correctness-cpu-offload`
- `basic-correctness-cumem`
- `basic-correctness-prefetch-offload`
- `basic-correctness-sleep-mode`
- `basic-models-test-other-cpu`
- `basic-models-tests-initialization`
- `basic-models-tests-other`
- `benchmarks-cli-test`
- `cpu-language-generation-and-pooling-model-tests` ×3
- `distributed-model-tests-2-gpus` ×3
- `entrypoints-integration-api-server` ×4
- `entrypoints-integration-api-server-generate`
- `entrypoints-integration-api-server-openai-chat_completion`
- `entrypoints-integration-api-server-openai-completion`
- `entrypoints-integration-llm`
- `entrypoints-integration-multimodal`
- `entrypoints-integration-pooling`
- `entrypoints-integration-responses-api`
- `entrypoints-integration-speech_to_text`
- `kernels-fla-ops-test-b200`
- `kernels-mhc-test-b200`
- `kernels-root-misc-test-b200`
- `language-models-tests-granite-l4-compatibility`
- `language-models-tests-hybrid` ×2
- `language-models-tests-standard`
- `multi-modal-models-standard-1-qwen2`
- `multi-modal-models-standard-2-qwen3-gemma`
- `multi-modal-models-standard-3-llava-qwen2-vl`
- `multi-modal-models-standard-4-other-whisper`
- `multi-modal-processor` ×4
- `multi-modal-processor-cpu` ×4
- `qwen4-exp-unit-tests`
- `v1-spec-decode`

</details>

<details><summary>Would add (today's rules do not run them) (0)</summary>

none

</details>

<details><summary>AMD mirrors: would skip (32)</summary>

- `basic-correctness` ×2
- `basic-correctness-cpu-offload`
- `basic-correctness-cumem`
- `basic-correctness-prefetch-offload`
- `basic-correctness-sleep-mode`
- `basic-models-tests-initialization`
- `basic-models-tests-other`
- `benchmarks-cli-test`
- `distributed-model-tests-2-gpus` ×3
- `entrypoints-integration-api-server` ×4
- `entrypoints-integration-api-server-generate`
- `entrypoints-integration-api-server-openai-chat_completion`
- `entrypoints-integration-api-server-openai-completion`
- `entrypoints-integration-llm`
- `entrypoints-integration-multimodal`
- `entrypoints-integration-pooling`
- `entrypoints-integration-responses-api`
- `entrypoints-integration-speech_to_text`
- `kernels-fla-ops-test-b200`
- `kernels-mhc-test-b200`
- `kernels-root-misc-test-b200`
- `language-models-tests-granite-l4-compatibility`
- `language-models-tests-hybrid` ×2
- `language-models-tests-standard`
- `multi-modal-models-standard-1-qwen2`
- `multi-modal-models-standard-2-qwen3-gemma`
- `multi-modal-models-standard-3-llava-qwen2-vl`
- `multi-modal-models-standard-4-other-whisper`
- `multi-modal-processor` ×4
- `platform-tests`
- `pytorch-compilation-passes-unit-tests`
- `v1-spec-decode`

</details>

<details><summary>AMD mirrors: would add (0)</summary>

none

</details>

<sub>2 changed files · base `1388100560` · head `9250458acf` · Python record: build 92817 at `7867d6c52d` · kernel record: table [`7867d6c`](https://github.com/aarnphm/aarnphm.github.io/commit/7867d6c52d) (build 92817), map [`7867d6c`](https://github.com/aarnphm/aarnphm.github.io/commit/7867d6c52d) · not counted: 10 build steps, 5 A100 steps the generator no longer emits, 3 optional steps the selector would also run</sub>

