---
title: '58489'
pageLayout: default
slug: thoughts/inference/vllm-prs/58489
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/58489.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#58489](https://github.com/aarnphm/aarnphm.github.io/issues/58489) \[Bugfix\]\[Qwen4Exp\] Keep pinned PLE prefetch ids out of the CUDA graph pool

[vllm-project/vllm#58489](https://github.com/vllm-project/vllm/pull/58489) · Juntian777 · MERGED · created 2026-09-24T01:47:40Z · merged 2026-09-25T02:51:51Z

## Problem and fix

Fixes [#58441](https://github.com/aarnphm/aarnphm.github.io/issues/58441). With `engram_config.cpu_offload=True`, `Qwen4ExpPLEPinnedHostEmbedding.start_prefetch` launches the PLE lookup on a side stream and returns; `_finalize_prefetch` joins it one decoder layer later. Qwen4Exp defaults to breakable CUDA graphs, so for PIECEWISE batches (prefill/mixed batches up to the capture size, or every batch with `cudagraph_mode=PIECEWISE`) `start_prefetch` is an eager break and its arguments are replaced by `weak_ref_tensor`s. Two things then combine:

- `gathered_ids.record_stream(prefetch_stream)` is a no-op: with `etp_data_parallel_size == 1`, `gathered_ids` is the weak ref itself, and the caching allocator skips `recordStream` for storage it does not own.
- The only strong reference to the ids is the local `ngram_ids` in `Qwen4ExpNGramEmbedding.start_prefetch`, which drops as soon as it returns, while the next segment is already being captured. The graph-pool block goes straight back to the free list and the rest of the preceding decoder layer can be captured on top of it.

At replay the side stream reads the ids while the main stream may be overwriting them, so greedy outputs silently stop being reproducible.

The fix computes the prefetch ids into a persistent per-layer buffer through the existing `output=` argument of `compute_ngram_ids`. This mirrors the persistent `_prefetch_buffer` already used for the lookup output, adds no kernel or copy, and costs `max_num_batched_tokens * ngram_heads * 8` bytes per PLE layer. FULL graphs and eager runs were already safe (the argument stays a real tensor there), and ETP spanning DP is unaffected (the ids are re-gathered into a fresh tensor inside the eager break).

## Validation



```bash
.venv/bin/python -m pytest tests/models/qwen4_exp/test_ple.py -q
.venv/bin/pre-commit run --files tests/models/qwen4_exp/test_ple.py vllm/models/qwen4_exp/nvidia/ngram_embedding.py
```

- **51 passed** on GB200. The new `test_ngram_prefetch_ids_outlive_start_prefetch` reproduces the lifetime bug without a full graph capture: the prefetch stub keeps a weak ref to the ids, as the eager break does, and a fresh `MemPool` makes the next same-size allocation reuse freed storage, as a later graph segment would. On main the weak ref reads the overwritten `-1`s (3/3 runs); with this change it reads the reference ids.
- Pre-commit passed.
- No model eval: the ids are produced by the same kernel into a different buffer, so outputs only change where the race corrupted them. The reporter’s cold/warm greedy table in [#58441](https://github.com/aarnphm/aarnphm.github.io/issues/58441) (8/8 identical once the ids are in a persistent buffer) covers the end-to-end effect.

## Related work and AI assistance

Open-PR search found no equivalent fix. [#57785](https://github.com/aarnphm/aarnphm.github.io/issues/57785) joins side-stream work during breakable capture and does not affect replay (row 2 of the table in [#58441](https://github.com/aarnphm/aarnphm.github.io/issues/58441)); [#57497](https://github.com/aarnphm/aarnphm.github.io/issues/57497) and [#58439](https://github.com/aarnphm/aarnphm.github.io/issues/58439) move or add PLE backends without changing this lifetime.

Claude Code assisted with root-causing, implementation and validation; the human submitter reviewed every changed line.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---

## comments

**github-actions** (2026-09-24T02:53:31Z):

✅ @Juntian777, CI is now available for this PR.

- `/ci run` starts upstream CI; `/amd-ci run` starts AMD CI only.
- Your branch must contain every commit currently on its upstream target branch. Merge or rebase onto the latest target branch, then rerun the command. Append `--allow-stale` to a run command to test an outdated branch at your own risk.
- `/ci retry` retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
- `/amd-ci retry` retries failed jobs in AMD CI for the current PR head. Use `/amd-ci run` when the current head has no AMD CI build.
- `/ci cancel` cancels scheduled or running CI builds for this PR branch; `/amd-ci cancel` does the same for AMD CI only.

<!-- vllm-ci-authorized -->

**ZJY0516** (2026-09-24T03:11:26Z):

/ci run —allow-stale

**github-actions** (2026-09-24T03:11:45Z):

✅ Triggered [Buildkite CI #90843](https://buildkite.com/vllm/ci/builds/90843) for commit `deeff0dd37b5`.

⚠️ This PR is 4 commits behind upstream `main`. Running CI at your own risk because `--allow-stale` was requested; outdated CI configuration may cause failures. Before merging, merge or rebase onto the latest `main`, then rerun `/ci run` on the latest PR commit.

**peakcrosser7** (2026-09-24T03:26:36Z):

Thanks for the fix! I hadn’t considered this tensor lifetime issue before.

**Juntian777** (2026-09-24T21:33:33Z):

/ci run

**github-actions** (2026-09-24T21:33:53Z):

✅ Triggered [Buildkite CI #91054](https://buildkite.com/vllm/ci/builds/91054) for commit `9f4d60f8c074`.

**Juntian777** (2026-09-24T23:07:47Z):

/ci run

**github-actions** (2026-09-24T23:08:07Z):

✅ Triggered [Buildkite CI #91081](https://buildkite.com/vllm/ci/builds/91081) for commit `866615c7035a`.

