---
title: '54517'
pageLayout: default
slug: thoughts/inference/vllm-prs/54517
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/54517.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#54517](https://github.com/aarnphm/aarnphm.github.io/issues/54517) \[Qwen3.8-Flash-Next\] Fuse Qwen4Exp PLE kernels

[vllm-project/vllm#54517](https://github.com/vllm-project/vllm/pull/54517) · gau-nernst · MERGED · created 2026-08-31T06:34:10Z · merged 2026-09-02T04:32:16Z

## Purpose

- Optimizations:
  - Fuse n-gram ID generation, dilated short-conv, and the PLE gate on NVIDIA
  - Merge key/value projections
- n-gram ID generation still has PyTorch fallback for future PLE offload ([#53899](https://github.com/aarnphm/aarnphm.github.io/issues/53899), not landed in main)
- Original PyTorch implementations of dilated short-conv (prefill, decode, spec-decode) are moved to `test_ple.py` for correctness testing. Production code only contains Triton kernel now.

### Correctness details

**n-gram ID generation**

- Preserve PyTorch int64 overflow behavior. Overflow hashes may become negative, so the kernel explicitly normalizes modulo results to match torch.remainder, guaranteeing every ID remains within its head-specific range: \[offset, offset + vocab_size).kernel
- Ragged request boundaries are resolved from query_start_loc, including empty requests and non-power-of-two request counts. Tokens preceding the current chunk come from persistent n-gram context, and crossing EOS replaces all older suffix positions with EOS.

**Dilated short-conv**

- Preserves cache semantics for decode, prefill, and speculative decode. Speculative state is rolled back by num_accepted_tokens - 1, while current candidate inputs are retained for the next target-model step.
- Convolution output and cache writeback use separate kernels so state is not overwritten before all causal reads complete.
- For mixed-spec/non-spec batches, token positions are looked up inside the kernel, eliminating separate gather/scatter ops.

**PLE gate**

- Match PyTorch numerics exactly (cast intermediates to BF16 as appropriately)

## Microbenchmarks

All results are obtained with GB300. Although we only show speedup here, another major benefit is greatly reduced memory usage as we don’t materialize temporary intermediates.

### N-gram IDs generation

| Workload            | Tokens | Requests |      Eager |  This PR | Speedup over eager |
| ------------------- | -----: | -------: | ---------: | -------: | -----------------: |
| Decode              |      1 |        1 |  76.448 us | 2.432 us |             31.43x |
| Decode              |     32 |       32 |  86.592 us | 3.040 us |             28.48x |
| Spec decode, 1 x 4  |      4 |        1 |  94.656 us | 2.464 us |             38.42x |
| Spec decode, 32 x 4 |    128 |       32 | 119.393 us | 3.232 us |             36.94x |
| Prefill, 1 x 8192   |  8,192 |        1 | 141.760 us | 3.584 us |             39.55x |
| Prefill, 2 x 8192   | 16,384 |        2 | 171.137 us | 4.192 us |             40.83x |
| Prefill, 4 x 4096   | 16,384 |        4 | 176.577 us | 4.416 us |             39.99x |

Eager is the baseline because N-gram IDs falls under custom op <span>&rarr;</span> does not go through torch.compile

### PLE gate

| Tokens |        Eager |   Compiled |    This PR | Speedup over compiled |
| -----: | -----------: | ---------: | ---------: | --------------------: |
|      1 |    86.657 us |   5.984 us |   4.064 us |                 1.47x |
|      4 |   111.360 us |  11.072 us |   3.552 us |                 3.12x |
|     16 |   110.544 us |  13.120 us |   3.776 us |                 3.47x |
|     64 |   104.288 us |  14.944 us |   4.544 us |                 3.29x |
|    256 |   157.313 us |  17.888 us |   7.808 us |                 2.29x |
|  1,024 |   365.457 us |  47.424 us |  21.280 us |                 2.23x |
|  4,096 | 1,567.925 us | 106.081 us |  76.896 us |                 1.38x |
| 16,384 | 5,949.587 us | 403.202 us | 289.825 us |                 1.39x |

### PLE short-conv

| Workload            | Tokens |        Eager |    This PR | Speedup over eager |
| ------------------- | -----: | -----------: | ---------: | -----------------: |
| Decode, 1 request   |      1 |    34.880 us |   5.984 us |              5.83x |
| Decode, 32 requests |     32 |    73.568 us |   7.328 us |             10.04x |
| Prefill, 1 x 8192   |  8,192 | 2,865.337 us | 355.681 us |              8.06x |
| Prefill, 2 x 8192   | 16,384 | 5,579.278 us | 703.490 us |              7.93x |
| Prefill, 4 x 4096   | 16,384 | 5,608.687 us | 724.546 us |              7.74x |
| Spec decode, 32 x 4 |    128 |   198.544 us |  16.448 us |             12.07x |
| Mixed batch         |  8,288 | 3,479.274 us | 384.737 us |              9.04x |

Eager is the baseline because short-conv falls under custom op <span>&rarr;</span> does not go through torch.compile

## Nsys trace

MTP3, TP4 on GB300

<img width="1544" height="694" alt="image" src="https://github.com/user-attachments/assets/79670d97-ecb7-4aea-8074-082ed1ff11df" />

## E2E perf

- TP4 GB300, Qwen/Qwen3.8-Flash-Next
- Baseline is [`003e343`](https://github.com/aarnphm/aarnphm.github.io/commit/003e34341a824ab8e3bec4861e92b5e84180af8b)



```
vllm serve {model}
--port {port}
-tp 4
--no-enable-prefix-caching
--no-enable-flashinfer-autotune
--enable-auto-tool-choice
--tool-call-parser qwen3_coder
--reasoning-parser qwen3
```



```
VLLM_USE_RUST_BENCH="1"

vllm bench serve
--backend openai-chat
--base-url http://{remote_host}:8000
--model {model}
--dataset-name speed-bench
--speed-bench-config throughput_8k
--speed-bench-max-input-len 8192
--speed-bench-output-len 1024
--ignore-eos
--num-warmups 5
--sweep-max-concurrency 1,4,16,64
--sweep-num-prompts-factor 10
--percentile-metrics "ttft,tpot,itl,e2el"
--metric-percentiles 50,90,99
--result-dir "{log_dir}/results"
--save-result
```

MTP3 `--speculative-config '{"method":"mtp","num_speculative_tokens":3}'`

| Concurrency | P50 TTFT before | P50 TTFT after     | P50 1/TPOT before | P50 1/TPOT after      |
| ----------- | --------------- | ------------------ | ----------------- | --------------------- |
| 1           | 184.31 ms       | 172.31 ms (−6.51%) | 234.00 tok/s      | 244.61 tok/s (+4.53%) |
| 4           | 216.74 ms       | 198.27 ms (−8.52%) | 170.58 tok/s      | 177.43 tok/s (+4.01%) |
| 16          | 223.82 ms       | 209.72 ms (−6.30%) | 97.39 tok/s       | 100.18 tok/s (+2.86%) |
| 64          | 245.79 ms       | 232.24 ms (−5.51%) | 46.00 tok/s       | 47.59 tok/s (+3.46%)  |

## Test Plan

### Unit test



```
pytest -v tests/models/qwen4_exp/test_ple.py
```

### E2E

TP4 on GB300

- GSM8K: think=false, max_tokens=1024, temp=0
- MMMU-Pro: think=true, max_tokens=65536

| Model | Mode    | GSM8K flexible / strict | MMMU-Pro flexible / strict |
| ----- | ------- | ----------------------- | -------------------------- |
| BF16  | non-MTP | 0.9659 / 0.9666         | 0.7653 / 0.7653            |
| BF16  | MTP3    | 0.9666 / 0.9666         | 0.7688 / 0.7688            |
| FP8   | non-MTP | 0.9682 / 0.9685         | 0.7665 / 0.7676            |
| FP8   | MTP3    | 0.9693 / 0.9701         | 0.7613 / 0.7613            |

## Test Result

---

<details>
<summary> Essential Elements of an Effective PR Description Checklist </summary>

- [ ] The purpose of the PR, such as “Fix some issue (link existing issues this PR will resolve)”.
- [ ] The test plan, such as providing test command.
- [ ] The test results, such as pasting the results comparison before and after, or e2e results
- [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.

</details>

---

## comments

**gcanlin** (2026-08-31T08:50:11Z):

We found the OOM bug when running the current main branch because of the fixed-size conv kernel. I took some benchmark on the varlen version here: [vllm-project/vllm#54525](https://github.com/vllm-project/vllm/pull/54525) and should fix the OOM bug now. But this PR has included [#54525](https://github.com/aarnphm/aarnphm.github.io/issues/54525) and it’s better. I will close my PR, just FYI.

**gau-nernst** (2026-09-02T02:41:56Z):

/ci run

**github-actions** (2026-09-02T02:42:14Z):

✅ Triggered [Buildkite CI #86758](https://buildkite.com/vllm/ci/builds/86758) for commit `72cada56262e`.

