---
title: '60142'
pageLayout: default
slug: thoughts/inference/vllm-prs/60142
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/60142.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#60142](https://github.com/aarnphm/aarnphm.github.io/issues/60142) \[Bugfix\] Fix Qwen4Exp FP8 PLE pinned lookup kernel failing to compile below SM89

[vllm-project/vllm#60142](https://github.com/vllm-project/vllm/pull/60142) · vadiklyutiy · MERGED · created 2026-10-05T23:02:12Z · merged 2026-10-06T01:56:45Z

## Overview

Fixes [#59868](https://github.com/aarnphm/aarnphm.github.io/issues/59868). On GPUs below SM89 (the report uses 4 × RTX 3090), Qwen4Exp with an FP8 E4M3FN PLE table and `--engram-config '{"cpu_offload": true}'` crashes in `profile_run`. The crash is a Triton `CompilationError` from `_lookup_ple_embedding_from_pinned_kernel`. This PR makes that kernel copy rows as raw bytes, so the FP8 type never enters its signature.

## Claims

- `_lookup_ple_embedding_from_pinned_kernel` now compiles and runs on sm_86 with an FP8 E4M3FN table, and it copies FP8 rows bit-exactly.
- On an A40, three existing FP8 pinned-lookup tests in `tests/models/qwen4_exp/test_ple.py` failed before this change. They now pass.
- BF16 lookups are unchanged: same bits, same speed.
- The module still imports when Triton is not installed.

## Validation

Hardware: a machine with 4 × NVIDIA A40 (sm_86). The reproducer uses one GPU. Software: vLLM commit `92044241a` (`0.30.1rc1.dev648+g92044241a`), with and without this PR, PyTorch 2.13.0+cu130, Triton 3.7.1, Python 3.12.12.

**Reproducer.** The reported path needs a checkpoint with FP8 PLE weights and a global scale. The public `Qwen/Qwen3.8-Flash-Next` checkpoint has BF16 PLE weights and no scale. So the reproducer launches the kernel directly with a small FP8 tensor. Below is the core of the script that ran. The full script only adds prints and a `try`/`except`.



```python
import torch
from vllm.models.qwen4_exp.common.ngram_embedding import (
  _lookup_ple_embedding_from_pinned_kernel,
)

N, D, BLOCK_D = 4, 16, 16
device = torch.device('cuda', 0)
weight = torch.zeros(N, D, dtype=torch.float8_e4m3fn, device=device)
ids = torch.tensor([0], dtype=torch.long, device=device)
output = torch.zeros(1, D, dtype=torch.bfloat16, device=device)

_lookup_ple_embedding_from_pinned_kernel[(ids.numel(),)](
  weight, ids, output, D, 0, N, BLOCK_D=BLOCK_D
)
torch.cuda.synchronize()
```

Run it from outside the source tree: `python -P repro_59868.py`.

\|        | Runs          | Result                          |
\| ------ | ------------- | ------------------------------- | --- | ----- | ------------- | ----------------------------- |
\| Before | 3 of 3 failed | `CompilationError`, exit code 1 |     | After | 2 of 2 passed | launch completes, exit code 0 |

Two of the “before” runs used the `vllm/vllm-openai:nightly-92044241a02f05de51420d654daf669de20b4691` image, and one used a source install. The “after” runs used the final patch. Three more runs on earlier versions of the patch also passed. Before the fix, every run showed the same error as the report:



```
triton.compiler.errors.CompilationError: at 1:0:
def _lookup_ple_embedding_from_pinned_kernel(
^
ValueError("type fp8e4nv not supported in this architecture. The supported fp8 dtypes are ('fp8e4b15', 'fp8e5')")
```

The reproducer writes from an FP8 weight into a BF16 output. vLLM never does this, because `_lookup` requires `output.dtype == weight.dtype`. After the fix, that call copies the raw FP8 bytes. So the reproducer only shows that the launch compiles. The tests below check the values on the real path.

**Existing tests.**



```
python -m pytest tests/models/qwen4_exp/test_ple.py -q --noconftest -k "pinned or uva"
```

`--noconftest` was used only because `tblib` was not installed locally.

- Before: `3 failed, 13 passed, 59 deselected`. All three fail with the error above: `test_ple_pinned_embedding_loads_on_cpu_and_looks_up_through_uva[True]`, `test_amd_pinned_embedding_output_written_under_compile[fp8]` and `test_amd_pinned_embedding_output_written_under_cudagraph_capture[fp8]`. They go through `_lookup` and check exact values.
- After: `16 passed, 59 deselected`. The full `test_ple.py` together with `tests/test_triton_utils.py` gives `88 passed`.

**Other checks.** These used one-off scripts that are not part of this PR.

- NVIDIA serving flow with FP8 on the A40 (`start_prefetch` → `forward` → `dequantize`): bit-exact against `weight[ids]` for row widths 3 and 160. With an ETP size of 2, rows owned by the other rank come out as zero bytes, in FP8 and in BF16.
- Large BF16 lookup with 160-element rows, compared row by row with `weight[ids]`: 7,000,000 and 8,388,608 rows gave 0 wrong rows, with byte offsets up to 2,684,354,240. Without the int64 row index (see Details), 7,000,000 rows hit `CUDA error: an illegal memory access was encountered`.
- BF16 speed against the old kernel, reading a pinned host table through UVA: time ratio ×0.993 to ×1.03 for row widths 64 to 1024 and for 160 (the real model’s width). Outputs were bit-identical.
- Import with Triton hidden (`sys.modules["triton"] = None`) passes. Without the placeholder change it fails with `AttributeError: module 'triton.language' has no attribute 'pointer_type'`.
- `pre-commit run --files` on both changed files passes.

**CI gap.** On NVIDIA, CI does not run the GPU tests in `test_ple.py`, and no CI runner is below SM89. So CI cannot catch this bug if it comes back. The runs above are the validation. The AMD MI355 job does run `test_ple.py`, so it covers the new byte copy on AMD.

## Details

**Root cause.** The kernel’s pointer parameters had no annotations, so Triton took their types from the tensors. An FP8 E4M3FN table became a `*fp8e4nv` parameter. Triton 3.7.1 accepts `fp8e4nv` only on compute capability 8.9 and higher. It rejects the type while it lowers the signature, before it compiles the body. That is why the error points at `1:0`, the `def` line. On the A40, a kernel with an empty body and one `float8_e4m3fn` pointer fails the same way. With `float8_e5m2` it compiles.

**Fix.** The kernel does no math on the values. Its only caller, `Qwen4ExpPLEPinnedHostEmbedding._lookup`, requires the output dtype to match the weight dtype. FP8 scaling happens later, in PyTorch (`dequantize`). So the kernel is a pure row gather. Both pointers are now annotated `tl.pointer_type(tl.uint8)`. The caller passes `row_bytes = embedding_dim * weight.element_size()` and sets `BLOCK_D = next_power_of_2(row_bytes)`. This works for FP8 and for BF16. `_reduce_etp_embeddings` in the same file already moves FP8 rows as raw bytes. The fix has to be in the signature, because the signature is where the error happens. A `uint8` pointer cast inside the body would come too late. A `.view(torch.uint8)` at the call site would fix `_lookup`, but not a direct launch with an FP8 tensor like the reproducer.

The fix also needs two more changes:

- `row_id` is now int64. Offsets are now in bytes, so for BF16 `row_id * row_bytes` is twice the old element offset. An int32 product would overflow at half as many rows as before. With `embedding_across_dp`, it would overflow at DP × `max_num_batched_tokens` ≥ 419,431 instead of 838,861 (for example, DP8 × 65,536). `tl.program_id(0).to(tl.int64)` is the usual vLLM idiom.
- `TritonLanguagePlaceholder` gets `uint8` and `pointer_type`. Python evaluates parameter annotations at import time, so without these names the module fails to import when Triton is missing. This is the first `tl.pointer_type` annotation in `vllm/`. mypy rejects it, so both lines carry `# type: ignore[valid-type]`.

**Limitations.**

- No end-to-end `vllm serve` run with an FP8 PLE checkpoint (see Reproducer). Only sm_86 was tested, not sm_80.
- `_engram_lookup_kernel` in `vllm/models/deepseek_v41/common/engram.py` also takes an FP8 E4M3FN table with no annotation, so it may fail the same way below SM89. It was not run. It does math on the values, so this byte-copy fix does not carry over, and it is not part of this PR.
- No model evaluation was run. The lookups are bit-exact in the tests above.

## E2E results (patch)

`vllm serve` with `-tp 4 --enable-expert-parallel --max-model-len 32768 --engram-config '{"cpu_offload": true}'`. The patch is applied on top of `vllm/vllm-openai:nightly-92044241a02f05de51420d654daf669de20b4691`. GSM8K is 5-shot, chat mode, thinking disabled (stderr ≈ ±0.7).

| Model                               | GPU          | PLE dtype             | Result                     | GSM8K flexible / strict |
| ----------------------------------- | ------------ | --------------------- | -------------------------- | ----------------------- |
| `Qwen/Qwen3.8-Flash-Next-FP8`       | A100 (SM80)  | FP8 E4M3, pinned host | starts and serves          | 92.95 / 91.96           |
| `Qwen/Qwen3.8-Flash-Next-FP8`       | B200 (SM100) | FP8 E4M3, pinned host | bit-identical to unpatched | 92.49 / 91.51           |
| `Inferact/Qwen3.8-Flash-Next-NVFP4` | B200 (SM100) | BF16, pinned host     | bit-identical to unpatched | 93.40 / 92.34           |

“Bit-identical” means greedy decoding at batch size 1: 10/10 prompts and 640/640 tokens match the unpatched build, with exactly equal logprobs.

---

<details>
<summary> Pull Request Checklist </summary>

- [x] I used vLLM’s `/pr-checklist` skill. (Mandatory for agents, optional for humans).

- [x] AI assistance was used during the creation of this PR.

- [x] **Design Fit:** Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

- [x] **Testing and Validation:** Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

- [x] **Code Quality and Style:** Keeps code and comments clear and concise, and updates relevant documentation and examples.

- [x] **Pull Request Contents:** Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide. </details>

---

## comments

**gau-nernst** (2026-10-06T00:29:57Z):

/ci run

**github-actions** (2026-10-06T00:30:17Z):

✅ Triggered [Buildkite CI #92992](https://buildkite.com/vllm/ci/builds/92992) for commit `cef0f0bced3d`.

**vllm-agent** (2026-10-06T01:01:17Z):

<!-- ci-selector-shadow -->

### CI selector (shadow): 16 test steps (24 jobs) instead of 46 (59 jobs)

Shadow mode: this changes nothing about what CI runs. It shows what the evidence-based selector would pick for this PR, next to today’s rules. [How it works](https://github.com/vllm-project/ci-infra/tree/main/buildkite/ci_selector).

**Feedback welcome:** reply here if it would skip a step this change needs, or runs something unrelated.

| steps (jobs)           | Today’s rules | Selector | Would skip | Would add |
| ---------------------- | ------------- | -------- | ---------- | --------- |
| NVIDIA, CPU and others | 46 (59)       | 16 (24)  | 42 (53)    | 12 (18)   |
| AMD mirrors            | 38 (46)       | 5 (5)    | 37 (45)    | 4 (4)     |

<details><summary>Selector would run (16)</summary>

- `arm-cpu-test` ×3
- `cpu-distributed-tests-dp-tp`
- `cpu-distributed-tests-pp-tp`
- `cpu-kernel-tests` ×2
- `cpu-language-generation-and-pooling-model-tests` ×3
- `cpu-multi-modal-model-tests-n` ×4
- `cpu-params-env-tokenizers-parser`
- `cpu-quantization-model-tests`
- `cpu-qwen2-5-vl-multimodal-tests`
- `cpu-spec-decode-tests`
- `e2e-scheduling-accuracy-1-gpu`
- `intel-hpu-test`
- `jit-monitor-no-runtime-jit`
- `qwen4-exp-unit-tests`
- `qwen4-exp-unit-tests-cpu`
- `v1-executor-worker`

</details>

<details><summary>Would skip (today's rules run them) (42)</summary>

- `ascend-npu-test`
- `basic-correctness` ×2
- `basic-correctness-cpu-offload`
- `basic-correctness-cumem`
- `basic-correctness-prefetch-offload`
- `basic-correctness-sleep-mode`
- `basic-models-test-other-cpu`
- `basic-models-tests-initialization`
- `basic-models-tests-other`
- `benchmarks-cli-test`
- `cpu-multimodal-config`
- `cpu-reasoning-renderers`
- `cpu-tool-parsers`
- `engine`
- `entrypoints-integration-api-server` ×4
- `entrypoints-integration-api-server-generate`
- `entrypoints-integration-api-server-openai-chat_completion`
- `entrypoints-integration-api-server-openai-completion`
- `entrypoints-integration-llm`
- `entrypoints-integration-multimodal`
- `entrypoints-integration-pooling`
- `entrypoints-integration-responses-api`
- `entrypoints-integration-speech_to_text`
- `kernels-fla-ops-test-b200`
- `kernels-mhc-test-b200`
- `kernels-root-misc-test-b200`
- `language-models-tests-granite-l4-compatibility`
- `language-models-tests-hybrid` ×2
- `language-models-tests-standard`
- `multi-modal-models-standard-1-qwen2`
- `multi-modal-models-standard-2-qwen3-gemma`
- `multi-modal-models-standard-3-llava-qwen2-vl`
- `multi-modal-models-standard-4-other-whisper`
- `multi-modal-processor` ×4
- `multi-modal-processor-cpu` ×4
- `pytorch-compilation-dynamic-shapes`
- `pytorch-compilation-passes-unit-tests`
- `pytorch-compilation-unit-tests`
- `pytorch-compilation-unit-tests-h100`
- `pytorch-fullgraph-cudagraph-l4-compatibility`
- `pytorch-fullgraph-test`
- `v1-spec-decode`

</details>

<details><summary>Would add (today's rules do not run them) (12)</summary>

- `arm-cpu-test` ×3 (code map)
- `cpu-distributed-tests-dp-tp` (code map)
- `cpu-distributed-tests-pp-tp` (code map)
- `cpu-kernel-tests` ×2 (code map)
- `cpu-multi-modal-model-tests-n` ×4 (code map)
- `cpu-quantization-model-tests` (code map)
- `cpu-qwen2-5-vl-multimodal-tests` (code map)
- `cpu-spec-decode-tests` (code map)
- `e2e-scheduling-accuracy-1-gpu` (code map)
- `intel-hpu-test` (code map)
- `jit-monitor-no-runtime-jit` (code map)
- `v1-executor-worker` (code map)

</details>

<details><summary>AMD mirrors: would skip (37)</summary>

- `basic-correctness` ×2
- `basic-correctness-cpu-offload`
- `basic-correctness-cumem`
- `basic-correctness-prefetch-offload`
- `basic-correctness-sleep-mode`
- `basic-models-tests-initialization`
- `basic-models-tests-other`
- `benchmarks-cli-test`
- `engine`
- `entrypoints-integration-api-server` ×4
- `entrypoints-integration-api-server-generate`
- `entrypoints-integration-api-server-openai-chat_completion`
- `entrypoints-integration-api-server-openai-completion`
- `entrypoints-integration-llm`
- `entrypoints-integration-multimodal`
- `entrypoints-integration-pooling`
- `entrypoints-integration-responses-api`
- `entrypoints-integration-speech_to_text`
- `kernels-fla-ops-test-b200`
- `kernels-mhc-test-b200`
- `kernels-root-misc-test-b200`
- `language-models-tests-granite-l4-compatibility`
- `language-models-tests-hybrid` ×2
- `language-models-tests-standard`
- `multi-modal-models-standard-1-qwen2`
- `multi-modal-models-standard-2-qwen3-gemma`
- `multi-modal-models-standard-3-llava-qwen2-vl`
- `multi-modal-models-standard-4-other-whisper`
- `multi-modal-processor` ×4
- `platform-tests`
- `pytorch-compilation-dynamic-shapes`
- `pytorch-compilation-passes-unit-tests`
- `pytorch-compilation-unit-tests`
- `pytorch-compilation-unit-tests-h100`
- `pytorch-fullgraph-cudagraph-l4-compatibility`
- `pytorch-fullgraph-test`
- `v1-spec-decode`

</details>

<details><summary>AMD mirrors: would add (4)</summary>

- `batch-invariance-h100` (code map)
- `docker-build-metadata` (code map)
- `engine-1-gpu` (code map)
- `v1-executor-worker` (code map)

</details>

<sub>2 changed files · base `b2863c0de9` · head `cef0f0bced` · Python record: build 92956 at `18c8a65edd` · kernel record: table [`18c8a65`](https://github.com/aarnphm/aarnphm.github.io/commit/18c8a65edd) (build 92956), map [`18c8a65`](https://github.com/aarnphm/aarnphm.github.io/commit/18c8a65edd) · not counted: 10 build steps, 5 A100 steps the generator no longer emits, 16 optional steps the selector would also run</sub>

**vadiklyutiy** (2026-10-06T01:35:23Z):

> LGTM. Didn’t know we can annotate the pointer type to change its type.

me too, before fixing it :)

