---
title: '58961'
pageLayout: default
slug: thoughts/inference/vllm-prs/58961
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/58961.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#58961](https://github.com/aarnphm/aarnphm.github.io/issues/58961) \[Bugfix\]\[Qwen4Exp\] Release the profiling KV cache held by QSA key views

[vllm-project/vllm#58961](https://github.com/vllm-project/vllm/pull/58961) · lucifer1004 · MERGED · created 2026-09-28T03:17:19Z · merged 2026-09-28T08:11:47Z

## Purpose

Fixes [#58960](https://github.com/aarnphm/aarnphm.github.io/issues/58960).

`QSAKeyStateCache.bind_kv_cache` stored the views `key_cache` and `rope_position_cache` next to `kv_cache`. `clear_layer_kv_caches` only resets `kv_cache`, so after CUDA graph memory profiling these views kept the profiling KV cache alive (9.76 GiB for Qwen3.8-Flash-Next-NVFP4 at TP2) past `_teardown_profiling_state()`’s `empty_cache()`. It was freed only in the real `initialize_kv_cache`, stayed cached, and a later long-lived allocation pinned the block. FULL CUDA graph capture then ran out of memory at `--gpu-memory-utilization 0.90` with 9.38 GiB free but unreleasable in the default pool. The issue has the memory-snapshot evidence.

The change derives both views from `kv_cache` on access, so `kv_cache` is the only reference to the bound storage, the same contract Mamba layers follow. Read sites (`indexer_qsa.py` on NVIDIA and AMD) are unchanged; they get the same tensor views as before.

**Not a duplicate:** no open PR or issue covers QSA cache lifetime or profiling teardown for Qwen4Exp. [#58439](https://github.com/aarnphm/aarnphm.github.io/issues/58439) and [#58815](https://github.com/aarnphm/aarnphm.github.io/issues/58815) change Qwen4Exp PLE storage, not the QSA cache; [#46515](https://github.com/aarnphm/aarnphm.github.io/issues/46515) fixes a different leak during CUDA graph memory profiling.

## Test Plan



```sh
# Regression test (new) and the rest of the QSA suite
pytest tests/models/qwen4_exp/test_qsa_reference.py -q

# End to end (2x RTX PRO 6000, TP2)
vllm serve nvidia/Qwen3.8-Flash-Next-NVFP4 --language-model-only \
  --tensor-parallel-size 2 --max-model-len 262144 \
  --gpu-memory-utilization 0.90 --reasoning-parser qwen3
```

## Test Result

- `test_clearing_qsa_key_cache_releases_its_storage` fails on `main` (the cleared layer still holds the bound storage) and passes with this change.
- `tests/models/qwen4_exp/test_qsa_reference.py`: 73 passed.
- `pre-commit` hooks (ruff-check, ruff-format, typos, mypy-3.10, check-test-tethering) pass on the changed files.
- End to end, same command, memory snapshots from the worker:

| default pool             | before                                            | after                                                               |
| ------------------------ | ------------------------------------------------- | ------------------------------------------------------------------- |
| free after KV cache init | 9.79 GiB (9,996 MiB profiling segment still held) | 0.02 GiB (segment released)                                         |
| free before capture      | 9.25 GiB, 9.23 GiB pinned in one segment          | 3.95 GiB, largest segment 0.82 GiB and fully free                   |
| FULL CUDA graph capture  | OOM in `qsa_select_paged_decode`                  | completes (`0.74 GiB (actual), 1.37 GiB (estimated)`), server ready |

Model evaluation: the change only affects when the profiling cache is freed; the views used at inference are the same tensors, so outputs are unchanged. For reference, the same model and stack at TP2 score 0.8931 on chat-template GSM8K (flexible-extract).

AI assistance was used (Claude) to investigate, write, and test this change.

---

## comments

**github-actions** (2026-09-28T06:57:23Z):

✅ @lucifer1004, CI is now available for this PR.

- `/ci run` starts upstream CI; `/amd-ci run` starts AMD CI only.
- Your branch must contain every commit currently on its upstream target branch. Merge or rebase onto the latest target branch, then rerun the command. Append `--allow-stale` to a run command to test an outdated branch at your own risk.
- `/ci retry` retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
- `/amd-ci retry` retries failed jobs in AMD CI for the current PR head. Use `/amd-ci run` when the current head has no AMD CI build.
- `/ci cancel` cancels scheduled or running CI builds for this PR branch; `/amd-ci cancel` does the same for AMD CI only.

<!-- vllm-ci-authorized -->

**ZJY0516** (2026-09-28T06:57:33Z):

/ci run —allow-stale

**github-actions** (2026-09-28T06:58:57Z):

✅ Triggered [Buildkite CI #91547](https://buildkite.com/vllm/ci/builds/91547) for commit `551b4ae8deb4`.

