---
title: '60023'
pageLayout: default
slug: thoughts/inference/vllm-prs/60023
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/60023.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#60023](https://github.com/aarnphm/aarnphm.github.io/issues/60023) \[Bugfix\]\[Qwen4Exp\] Honor —kv-cache-dtype-skip-layers in QSA attention

[vllm-project/vllm#60023](https://github.com/vllm-project/vllm/pull/60023) · stecasta · MERGED · created 2026-10-05T07:24:37Z · merged 2026-10-05T12:13:32Z

## Purpose

`Qwen4ExpQSAAttention` takes its KV cache dtype straight from `cache_config.cache_dtype`, so
`--kv-cache-dtype-skip-layers` is silently ignored on every Qwen4Exp full-attention layer, the MTP
draft layer included. This resolves the dtype per layer, the same way `Attention` and `MLAAttention`
already do (`vllm/model_executor/layers/attention/{attention,mla_attention}.py`).

Why it matters: with `--kv-cache-dtype fp8`, FP8 KV in the MTP layer’s own attention lowers draft
acceptance at depth on Qwen3.8-Flash-Next. Skipping just that layer (`--kv-cache-dtype-skip-layers 48`)
keeps the target KV in FP8 and restores acceptance.

## Evidence

DGX Spark (GB10), Qwen3.8-Flash-Next NVFP4 @[`ada4da3`](https://github.com/aarnphm/aarnphm.github.io/commit/ada4da32), MTP with 6 drafts, greedy, 41 prompts,
c=1, one node, with this change applied (measured on a nightly [`0cbac6c`](https://github.com/aarnphm/aarnphm.github.io/commit/0cbac6cd1)-based image; the change is the same two lines on current main):

| target KV | MTP-layer KV                             | AL    | decode tok/s |
| --------- | ---------------------------------------- | ----- | ------------ |
| fp8       | fp8                                      | 3.215 | 43.3         |
| bf16      | fp8                                      | 3.178 | 43.7         |
| fp8       | bf16 (`--kv-cache-dtype-skip-layers 48`) | 3.473 | 48.7         |
| bf16      | bf16                                     | 3.504 | 50.0         |

With FP8 KV in the MTP layer, conditional acceptance at depth 2 to 6 is 0.66 to 0.72. With BF16 it
is 0.74 to 0.80. Depth 1 is unchanged.

At 32k context (8 prompts of 32,928 tokens, 6 GiB KV), the skip costs 7% of the KV pool
(211,502 to 196,608 tokens at 3 drafts). Converting the whole cache to BF16 costs 29%.

## Duplicate check

- PRs and issues searched: “kv-cache-dtype-skip-layers”, “kv_cache_dtype_skip_layers”,
  “Qwen4Exp kv cache dtype”, “QSA fp8 KV”, “MTP fp8 kv cache acceptance”, and open PRs touching
  qsa.py. None changes how QSA picks its per-layer KV dtype.
- Related, but not overlapping:
  - [#54846](https://github.com/aarnphm/aarnphm.github.io/issues/54846): fp8/nvfp4 KV read path for QSA.
  - [#56116](https://github.com/aarnphm/aarnphm.github.io/issues/56116): ModelOpt layer-wise mixed KV.
  - [#47618](https://github.com/aarnphm/aarnphm.github.io/issues/47618): v1/v2 runner dtype sync.

## Test Plan

- `pytest tests/models/qwen4_exp/test_qsa_reference.py -k skip_layers`
  - New regression test: the MTP layer prefix resolves to `auto` when listed, and stays `fp8` otherwise.
- End to end on GB10 as in the table above (`--kv-cache-dtype fp8 --kv-cache-dtype-skip-layers 48`).

## Test Result

- `pytest tests/models/qwen4_exp/test_qsa_reference.py` on this branch (rebased on main [`042ab03`](https://github.com/aarnphm/aarnphm.github.io/commit/042ab0305c)): 18 passed, 56 skipped (GPU-only cases).
- pre-commit clean.

AI assistance was used for this change. The submitter reviewed every line and ran the tests above.

---

## comments

**gau-nernst** (2026-10-05T11:07:03Z):

/ci run

**github-actions** (2026-10-05T11:07:24Z):

✅ Triggered [Buildkite CI #92872](https://buildkite.com/vllm/ci/builds/92872) for commit `f02322058e80`.

**vllm-agent** (2026-10-05T11:24:45Z):

<!-- ci-selector-shadow -->

### CI selector (shadow): 1 test steps (1 jobs) instead of 36 (51 jobs)

Shadow mode: this changes nothing about what CI runs. It shows what the evidence-based selector would pick for this PR, next to today’s rules. [How it works](https://github.com/vllm-project/ci-infra/tree/main/buildkite/ci_selector).

**Feedback welcome:** reply here if it would skip a step this change needs, or runs something unrelated.

| steps (jobs)           | Today’s rules | Selector | Would skip | Would add |
| ---------------------- | ------------- | -------- | ---------- | --------- |
| NVIDIA, CPU and others | 36 (51)       | 1 (1)    | 35 (50)    | 0 (0)     |
| AMD mirrors            | 33 (43)       | 0 (0)    | 33 (43)    | 0 (0)     |

<details><summary>Selector would run (1)</summary>

- `qwen4-exp-unit-tests`

</details>

<details><summary>Would skip (today's rules run them) (35)</summary>

- `ascend-npu-test`
- `basic-correctness` ×2
- `basic-correctness-cpu-offload`
- `basic-correctness-cumem`
- `basic-correctness-prefetch-offload`
- `basic-correctness-sleep-mode`
- `basic-models-test-other-cpu`
- `basic-models-tests-initialization`
- `basic-models-tests-other`
- `benchmarks-cli-test`
- `cpu-language-generation-and-pooling-model-tests` ×3
- `distributed-model-tests-2-gpus` ×3
- `entrypoints-integration-api-server` ×4
- `entrypoints-integration-api-server-generate`
- `entrypoints-integration-api-server-openai-chat_completion`
- `entrypoints-integration-api-server-openai-completion`
- `entrypoints-integration-llm`
- `entrypoints-integration-multimodal`
- `entrypoints-integration-pooling`
- `entrypoints-integration-responses-api`
- `entrypoints-integration-speech_to_text`
- `kernels-fla-ops-test-b200`
- `kernels-mhc-test-b200`
- `kernels-root-misc-test-b200`
- `language-models-tests-granite-l4-compatibility`
- `language-models-tests-hybrid` ×2
- `language-models-tests-standard`
- `multi-modal-models-standard-1-qwen2`
- `multi-modal-models-standard-2-qwen3-gemma`
- `multi-modal-models-standard-3-llava-qwen2-vl`
- `multi-modal-models-standard-4-other-whisper`
- `multi-modal-processor` ×4
- `multi-modal-processor-cpu` ×4
- `qwen4-exp-unit-tests-cpu`
- `v1-spec-decode`

</details>

<details><summary>Would add (today's rules do not run them) (0)</summary>

none

</details>

<details><summary>AMD mirrors: would skip (33)</summary>

- `basic-correctness` ×2
- `basic-correctness-cpu-offload`
- `basic-correctness-cumem`
- `basic-correctness-prefetch-offload`
- `basic-correctness-sleep-mode`
- `basic-models-tests-initialization`
- `basic-models-tests-other`
- `benchmarks-cli-test`
- `distributed-model-tests-2-gpus` ×3
- `entrypoints-integration-api-server` ×4
- `entrypoints-integration-api-server-generate`
- `entrypoints-integration-api-server-openai-chat_completion`
- `entrypoints-integration-api-server-openai-completion`
- `entrypoints-integration-llm`
- `entrypoints-integration-multimodal`
- `entrypoints-integration-pooling`
- `entrypoints-integration-responses-api`
- `entrypoints-integration-speech_to_text`
- `kernels-fla-ops-test-b200`
- `kernels-mhc-test-b200`
- `kernels-root-misc-test-b200`
- `language-models-tests-granite-l4-compatibility`
- `language-models-tests-hybrid` ×2
- `language-models-tests-standard`
- `multi-modal-models-standard-1-qwen2`
- `multi-modal-models-standard-2-qwen3-gemma`
- `multi-modal-models-standard-3-llava-qwen2-vl`
- `multi-modal-models-standard-4-other-whisper`
- `multi-modal-processor` ×4
- `platform-tests`
- `pytorch-compilation-passes-unit-tests`
- `qwen4-exp-unit-tests`
- `v1-spec-decode`

</details>

<details><summary>AMD mirrors: would add (0)</summary>

none

</details>

<sub>2 changed files · base `710ac56e69` · head `f02322058e` · Python record: build 92817 at `7867d6c52d` · kernel record: table [`7867d6c`](https://github.com/aarnphm/aarnphm.github.io/commit/7867d6c52d) (build 92817), map [`7867d6c`](https://github.com/aarnphm/aarnphm.github.io/commit/7867d6c52d) · not counted: 10 build steps, 5 A100 steps the generator no longer emits, 2 optional steps the selector would also run</sub>

