---
title: '58114'
pageLayout: default
slug: thoughts/inference/vllm-prs/58114
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/58114.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#58114](https://github.com/aarnphm/aarnphm.github.io/issues/58114) \[Perf\]\[Qwen3.8\] Reduce PLE metadata construction overhead

[vllm-project/vllm#58114](https://github.com/vllm-project/vllm/pull/58114) · ZJY0516 · MERGED · created 2026-09-22T08:19:05Z · merged 2026-09-28T12:46:23Z

## Purpose

PLE metadata construction still synchronizes in mixed speculative batches and builds temporary tensors that its consumers do not need. This reduces per-step builder work while preserving request ordering, accepted-token routing, and persistent CUDA graph buffers.

- Supply `repeat_interleave` with the output size from CPU query offsets, excluding token padding, and use `index_fill_` for request-group labels to avoid scalar-assignment synchronization.
- In pure speculative batches, use request-prefix views instead of uploading indices and gathering state/acceptance tensors. Skip the unused device query-length calculation and identity token maps; request padding remains trailing.
- Let PLE skip generic causal-conv prefill metadata that neither PLE implementation consumes. Other Mamba builders retain the existing default.
- Reuse the builder’s prefill query offsets in NVIDIA and AMD PLE consumers.

Related to [#55922](https://github.com/aarnphm/aarnphm.github.io/issues/55922). This follows the already-merged [#55054](https://github.com/aarnphm/aarnphm.github.io/issues/55054), which made request-index uploads asynchronous; the changes here remove additional work and mixed-batch synchronization. I checked the tracking issue and its comments, open PRs referencing it, and searches for `PLE metadata`, `short_conv_attn`, and `PleShortConvAttentionMetadataBuilder`. No open PR implements this change. The related CPU backend, quantization, QSA, and standalone PLE state-kernel PRs address different paths.

## Test Plan



```bash
.venv/bin/python -m pytest \
  tests/models/qwen4_exp/test_ple.py \
  tests/v1/attention/test_mamba_update_block_table.py -q

.venv/bin/pre-commit run --files \
  tests/models/qwen4_exp/test_ple.py \
  vllm/models/qwen4_exp/amd/ple_layer.py \
  vllm/models/qwen4_exp/nvidia/ple_layer.py \
  vllm/v1/attention/backends/mamba_attn.py \
  vllm/v1/attention/backends/short_conv_attn.py

git diff --check
```

New builder coverage exercises CPU/CUDA, eager/full-graph buffer preparation, request and token padding, interleaved speculative/non-speculative requests, accepted-token routing, and prefill offsets. CUDA synchronization debug mode is enabled during CUDA builder calls.

Before marking ready: complete human line-by-line review and required test review. The GB200 end-to-end smoke/performance results are below; AMD runtime validation and broader model evaluation remain outstanding.

## Test Result

- GPU validation through Vigil/Slurm on GB200: **69 passed** across the two test files.
- Pre-commit hooks and `git diff --check`: **passed**.
- Relevant output tensors matched the original builder in the synthetic A/B cases below.

### Metadata builder microbenchmark

Baseline: `986e217158a9ed54eca8307c6bb9abac7c5b2278`. Same GB200 node and Python/PyTorch environment; original builder loaded alongside the patched builder. Three speculative tokens, full-graph buffer preparation, one CPU thread, 30 warmup builds per variant, then seven alternating rounds of 200 builds each. CUDA synchronization brackets each timed round; the computed-token cache is reset before each build. Values are median amortized host+device wall time per build.

| Batch                          | Base (us) | Patched (us) | Reduction |
| ------------------------------ | --------: | -----------: | --------: |
| Pure MTP, 1 request + padding  |    395.10 |       244.25 |    38.18% |
| Pure MTP, 8 requests + padding |    390.80 |       244.33 |    37.48% |
| Pure MTP, 32 requests          |    364.97 |       217.75 |    40.34% |
| Mixed spec/decode/1K prefill   |    994.08 |       678.51 |    31.75% |
| Non-spec decode + 1K prefill   |    414.81 |       215.74 |    47.99% |

These are metadata-builder measurements, not end-to-end throughput gains.

### End-to-end validation

Qwen3.8-Flash-Next-FP8 on one exclusive 4-GPU GB200 node via Vigil/Slurm: TP4, MTP3, prefix caching enabled, default serving capacities/backends/CUDA graph behavior. Baseline main `f92b78f6ef9c9b28f60668da77af5b65645b1a45`; patch `df5e6d382bb4cf80f4dc46073d546b44816c0956`. Both use the same checkout and binaries, with the four changed implementation modules imported from audited immutable source snapshots. FlashInfer Python/cubin both `0.6.18.post1`; no version-check bypass.

Four independent server starts in ABBA order (two per variant), three benchmark rounds per concurrency per start. Rust `vllm-bench`, exact 1024 input token IDs / 128 output tokens, temperature 0, ignore_eos, 8 warmup requests per invocation; 32 measured requests at C1 and 128 at C8. Matching seeds across starts (`4201/4301/4401` for C1, `4208/4308/4408` for C8). The installed client was separately checked to generate identical request bodies across repeated same-seed invocations.

**All 1,920 measured requests succeeded**, with exact input/output lengths. No prefix-cache hits during these performance runs. Throughput is total generated tokens divided by summed measured durations; TPOT excludes TTFT.

| Concurrency | Metric          | Baseline |   Patch | Change |
| ----------- | --------------- | -------: | ------: | -----: |
| 1           | Output tokens/s |  244.326 | 240.623 | -1.52% |
| 1           | Mean TPOT (ms)  |    3.635 |   3.710 | +2.07% |
| 1           | Mean TTFT (ms)  |   62.158 |  60.625 | -2.47% |
| 8           | Output tokens/s |  489.977 | 514.283 | +4.96% |
| 8           | Mean TPOT (ms)  |   14.040 |  13.380 | -4.70% |
| 8           | Mean TTFT (ms)  |  287.536 | 274.665 | -4.48% |

Independent-start throughput (tokens/s):

| Start |     C1 |     C8 |
| ----- | -----: | -----: |
| A1    | 243.71 | 486.03 |
| B1    | 238.98 | 513.06 |
| B2    | 242.29 | 515.51 |
| A2    | 244.94 | 493.99 |

MTP acceptance lengths (baseline → patch): C1: 2.4496 → 2.3898, C8: 2.4693 → 2.4572.

Interpretation: the measured C8 workload improves across both paired starts; C1 does not demonstrate a reliable throughput gain. There are only two independent starts per variant, and output text/MTP acceptance varies even between same-version restarts. These are workload-specific observations, not a statistical confidence claim or a general serving-speedup guarantee.

GSM8K five-shot, 64 examples, thinking disabled, temperature 0, concurrency 4, max_tokens 4096: **baseline 63/64 and patch 63/64 (98.44%)**, zero request errors. All 64 per-question scores match, including the same incorrect example. Evaluations ran after performance measurements. This is an accuracy smoke test, not a full model evaluation.

Engine command used by Vigil:



```bash
vllm serve MODEL --served-model-name ple-metadata --port 8188 -tp 4 \
  --enable-prefix-caching \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'
```

Client command for the first C1 round (C8 uses `--num-prompts 128 --max-concurrency 8 --seed 4208`; later rounds increment the seed by 100):



```bash
vllm-bench --base-url http://HOST:8188 --model ple-metadata --tokenizer MODEL \
  --dataset-name random --random-input-len 1024 --random-output-len 128 \
  --random-range-ratio 0 --prompt-token-ids \
  --num-prompts 32 --max-concurrency 1 --num-warmups 8 \
  --ignore-eos --temperature 0 --seed 4201 \
  --save-result --save-detailed --result-dir RESULTS --result-filename r1-c1.json \
  --percentile-metrics ttft,tpot,itl,e2el --metric-percentiles 50,95,99
```

Raw per-request data, server counters, configurations, source/binary hashes, and all 24 round results are retained in the local `ple_metadata_e2e_20260922` artifact directory.

## AI Assistance

OpenAI Codex assisted with implementation, testing, benchmark analysis, and drafting this PR. Human review is still required before marking it ready. GB200 end-to-end performance and accuracy smoke validation are reported above; AMD runtime coverage remains outstanding.

---

<details>
<summary>Essential Elements of an Effective PR Description Checklist</summary>

- [x] Purpose and related work described.
- [x] Test commands and coverage provided.
- [x] Available results and end-to-end validation limitations reported.
- [x] No user-facing documentation change is needed for this internal optimization.

</details>

---

## comments

**mergify** (2026-09-24T22:14:09Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @ZJY0516.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**ZJY0516** (2026-09-28T10:55:22Z):

/ci run —allow-stale

**github-actions** (2026-09-28T10:55:43Z):

✅ Triggered [Buildkite CI #91575](https://buildkite.com/vllm/ci/builds/91575) for commit `1bc9e6bed9a8`.

⚠️ This PR is 6 commits behind upstream `main`. Running CI at your own risk because `--allow-stale` was requested; outdated CI configuration may cause failures. Before merging, merge or rebase onto the latest `main`, then rerun `/ci run` on the latest PR commit.

