---
title: '57105'
pageLayout: default
slug: thoughts/inference/vllm-prs/57105
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/57105.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#57105](https://github.com/aarnphm/aarnphm.github.io/issues/57105) \[Qwen3.8-Flash-Next\] Avoid memory fragmentation in QSA indexer logits workspace

[vllm-project/vllm#57105](https://github.com/vllm-project/vllm/pull/57105) · gau-nernst · MERGED · created 2026-09-16T03:09:05Z · merged 2026-09-27T02:11:53Z

## Purpose

Fixes [#56457](https://github.com/aarnphm/aarnphm.github.io/issues/56457)

Now we always reserve the worst-case logits workspace so that subsequent calls won’t hit a new allocation.

## Test Plan

## Test Result

---

<details>
<summary> Essential Elements of an Effective PR Description Checklist </summary>

- [ ] The purpose of the PR, such as “Fix some issue (link existing issues this PR will resolve)”.
- [ ] The test plan, such as providing test command.
- [ ] The test results, such as pasting the results comparison before and after, or e2e results
- [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.

</details>

---

## comments

**jahnclawdmonet** (2026-09-16T04:14:02Z):

Measured this on one GB10 (sm_121), TP=1, by calling `qsa_select_paged_prefill` directly and stepping `max_seq_len` through the 3200-token chunk boundaries of [#56457](https://github.com/aarnphm/aarnphm.github.io/issues/56457). No engine, so this neither reproduces nor clears the hang there, which is reported under TP=2 on two nodes.

Build: `vllm/vllm-openai@sha256:b0501f99fec5136f248f78d5850977a2ec32d55cd9a665f4a9ffef24cbdf7fe5`, vLLM 0.28.1rc1.dev580+g385dce36b, the official nightly at `385dce36bcee42309924a5ece951a96db3dce7f2`, 357 commits before this PR’s base. Nothing in that range touches `qsa_indexer.py`, so the image’s copy is the PR’s pre-image: `git hash-object` returns `9ddd65d15935ff6e97e2d68e826fa47e682c848b`, and the patched copy returns `9f7346198a2451929df826f9e985ecb445c551ff`. Everything else in the install sits at that older commit. torch 2.13.0+cu130.

From `RadixArk/Qwen3.8-Flash-Next-NVFP4`: `indexer_n_heads` 4, `indexer_head_dim` 128, `indexer_compress_ratio` 4, `indexer_budget` 2048. Chosen to match [#56457](https://github.com/aarnphm/aarnphm.github.io/issues/56457): 3200 query rows, compressed page size 800 (its 3200-token KV block over `compress_ratio` 4), 52 steps at 3200, 6400 … 166400, page table 52 pages wide, which is what 166,400 tokens needs and less than the 82 a 262,144 deployment would have. `VLLM_SPARSE_INDEXER_MAX_LOGITS_MB` left at 512. The compressed indexer cache here is bf16. Each arm walks all 52 lengths in one process, two calls per length, so a row is cumulative rather than an independent measurement.

| max_seq_len | logits_width | stock             | patched        |
| ----------- | ------------ | ----------------- | -------------- |
| 3200        | 832          | 48 MiB, 4 seg     | 534 MiB, 3 seg |
| 12800       | 3200         | 138 MiB, 7 seg    | 534 MiB, 3 seg |
| 38400       | 9600         | 812 MiB, 15 seg   | 534 MiB, 3 seg |
| 83200       | 20800        | 3494 MiB, 29 seg  | 534 MiB, 3 seg |
| 128000      | 32000        | 8092 MiB, 43 seg  | 534 MiB, 3 seg |
| 166400      | 41600        | 13556 MiB, 55 seg | 534 MiB, 3 seg |

Every call is a single chunk: `rows_per_chunk` stays above 3200 until the width passes 41,943, one step past the end of the walk. That is the condition for the per-step allocation to keep growing.

Both arms start at 22 MiB reserved, in two segments of 20 and 2 MiB. Stock ends at 55 segments, 54 of them 8 MiB or larger, the largest being 508, 500, 490, 480, 470, 460, 450, 440, 430, 422, 412 and 402 MiB. Every measured row is 22 plus 14 plus the running sum of `round(3200 * width * 4)` to 2 MiB, exact at all 52 steps; the 14 MiB is the benchmark’s own `torch.sort` index buffer, not the function under test. The logits part alone reaches 13,520 MiB, or 14.18 GB, against the 14 GB the issue estimates from `sum 10.24 MB * i` over 52 chunks. Patched ends at three segments, 512, 20 and 2 MiB, and reserves the 512 MiB workspace on its first call at 3200 tokens, so `VLLM_SPARSE_INDEXER_MAX_LOGITS_MB` still bounds the per-chunk slice but its value also becomes the floor on reserved memory. Neither arm recorded an allocator retry.

I hashed the `block_indices` buffer after each call, both as returned and with each row sorted. At 47 of the 52 lengths the two calls in each arm gave one row-sorted hash and the two arms gave the same one. At 41600, 83200 and 89600 stock gave two and patched gave one of stock’s two; at 70400 and 92800 each arm gave a single hash and the two differed. Two calls cannot enumerate an arm’s values, so I reran 41600, 70400 and 92800 at 24 calls per arm: each arm produced two distinct row-sorted hashes over its 24 calls and both arms produced the same two at all three lengths. The as-returned hash differed between the two calls at every one of the 52 lengths in both arms, and all 24 were distinct in both arms at the three reruns, so row order is unstable in both builds.

At 70400, 83200 and 89600 `visible_blocks` equals `logits_width`, so the kernel writes every column and no retained workspace content can reach the top-k; stock varies at those lengths anyway. At 41600 and 92800 the last 32 columns go unwritten. I did not determine the source of the variation.

Two things this setup does not cover. `visible_blocks` is one constant for all 3200 rows rather than growing with row position, so the unwritten tail is at most 32 columns here and zero at 26 of the 52 lengths, where a real chunk leaves most of each early row unwritten. And since every call is a single chunk, the path where a second chunk reads a workspace still holding the first chunk’s scores never runs. The query and cache tensors are `randn`, there is a single request and no decode rows, and I did not measure latency.

**gau-nernst** (2026-09-26T13:32:28Z):

/ci run

**github-actions** (2026-09-26T13:32:47Z):

✅ Triggered [Buildkite CI #91401](https://buildkite.com/vllm/ci/builds/91401) for commit `b729d8392768`.

**gau-nernst** (2026-09-26T15:19:19Z):

/ci retry

**github-actions** (2026-09-26T15:19:37Z):

✅ Triggered [Buildkite CI #91409](https://buildkite.com/vllm/ci/builds/91409) for commit `3a0340fd4329`, running 1 failed step(s) from [Buildkite CI #91401](https://buildkite.com/vllm/ci/builds/91401).

