---
title: Qwen3.8-Next serving notes
tags:
  - inference
  - vllm
  - new-architectures
pageLayout: default
slug: thoughts/inference/qwen3.8-serving
permalink: https://aarnphm.xyz/thoughts/inference/qwen3.8-serving.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# Qwen3.8-Next serving notes

Snapshot: vLLM main on 2026-10-11. Paths and PRs refer to that tree. Model-level architecture and code shape are in [[thoughts/inference/qwen3.8-runtime|Qwen3.8-Next runtime notes]].

## 3. KV cache & state (what lives where, per request)

**Per-layer specs** (TP=1, bf16; per-rank sizes shrink with TP except where marked replicated):

| Owner                                                        | Spec                                                                                         | Content                                                                                                                                  | Size                                                                              | Prefix-cacheable                               |
| ------------------------------------------------------------ | -------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | ---------------------------------------------- |
| QSA main KV (`layers.N.self_attn.attn`)                      | `FullAttentionSpec` (qsa.py:543-551)                                                         | K,V of 2 heads × 256                                                                                                                     | 2 KiB/token (1 KiB fp8)                                                           | yes                                            |
| QSA compressed index key (`...indexer.compressed_key_cache`) | `MLAAttentionSpec(tokens_per_state=4)` (qsa_cache.py:865-876)                                | RMSNorm+RoPE pooled key, 1 row per closed 4-token group, **TP-replicated** (ReplicatedLinear)                                            | 256 B / 4 tokens = 64 B/token (32 B fp8)                                          | yes; follows the main block table              |
| QSA raw-key ring (`...indexer.raw_key_cache`)                | **`CircularBufferSpec`** (qsa_cache.py:844-862; class at `v1/kv_cache_interface.py:909-947`) | raw (pre-norm) keys of the open group, plus 3×int64 exact MRoPE positions packed after the key when MRoPE is used (qsa_cache.py:811-842) | capacity = `c·⌈(c+num_spec)/c⌉` slots, **1 block per request for its whole life** | **no** (`prefix_cacheable=False`)              |
| GDN (36 layers)                                              | `MambaSpec` (GDN_ATTN)                                                                       | conv \[10240, 3+num_spec\] + SSM \[48, 128, 128\]                                                                                        | ≈1.5 MiB bf16 SSM + 80 KiB conv per layer \[derived\]                             | via the mamba cache mode (align / checkpoints) |
| PLE short conv (1 layer)                                     | `MambaSpec(SHORT_CONV, tp_replicated=True)` (model.py:822-827)                               | conv history \[10240, 9+num_spec\]                                                                                                       | 200 KiB @ num_spec=1, **replicated on every TP rank**                             | Mamba-style                                    |
| PLE table                                                    | not KV; a weight                                                                             | 51B params, pinned host                                                                                                                  | ≈51 GB fp8 / 102 GB bf16 per ETP group                                            | —                                              |

**Cache invariants and the layout planner**

- **Ring sizing.**
  - It must hold the committed members of the open group plus every draft row written before acceptance. Too small, and a rejected draft overwrites a committed key that the next step still needs to close the group (qsa_cache.py:845-851; tests `test_qsa_circular_buffer_survives_one_speculative_step`).
  - Its capacity must **divide the attention block_size**, because it joins the LCM that sets the scheduler block size (qsa_cache.py:852-855).
  - Examples: num_spec=1 gives 8 slots; num_spec=5 gives 12 slots, which forces block_size to be a multiple of 12 \[derived\].
- **Ring slot mapping:** `slot = block_table[req,0]·cap + pos % cap`. Only the last `cap` rows of each request’s chunk are written (`token_idx + cap >= query_end`). Requests on the null block (block 0) are skipped (qsa_cache.py:108-158, 281-297; [#59528](https://github.com/aarnphm/aarnphm.github.io/issues/59528)).
- **Compressed slot mapping:** written only at group-closing positions `(pos+1)%c==0`. It is masked by the main slot_mapping < 0 (qsa_cache.py:161-189, 298-320).
- **Packing.**
  - `QSAStateBackend.supported_kv_cache_layouts = (BLNHC, BLHNC)`. The reason given: “QSA pages are packed beside the main KV pages within each block” (qsa_cache.py:753-756).
  - Mixed page sizes force a block-compact layout, and with more than one group, a block-outermost one (`validate_kv_cache_layout`, `v1/core/kv_cache_utils.py:1555-1585`, `_get_packed_kv_cache_groups` :1860+).
  - This is the **BLHNC-vs-LBHNC layout RFC [#42082](https://github.com/aarnphm/aarnphm.github.io/issues/42082)** (`v1/kv_cache_layout.py:11-28`) in practice. Qwen4Exp _requires_ B-outer layouts.
- **Mamba page size.** Two Mamba specs (GDN, PLE) are present, and the platform takes the max page size (`vllm/platforms/interface.py:1044-1050`).
  - The PLE page is TP-replicated while GDN shrinks with TP.
  - \[derived\] At TP=8 with bf16 SSM: GDN ≈ 192 KiB + 15 KiB versus PLE 200–240 KiB. So **PLE becomes the largest page** at high TP or high num_spec.

**Prefix caching**

- `CircularBufferManager` claims exactly 1 block per request. It reports a cache hit of 0, and its cache and skip hooks do nothing (`v1/core/single_type_kv_cache_manager.py:1206-1295`).
- The coordinator excludes non-cacheable groups from hit lookup and from block-size divisibility (`v1/core/kv_cache_coordinator.py:670-679, 730-740`).
- **Why this is correct \[infer\]:** hits are aligned to block_size, which is a multiple of the ring capacity and so of c. At a hit boundary the open group is empty, so the compressor never needs ring rows from before the hit.
- GDN state needs mamba-cache-mode align or checkpoints. The ROCm eval config uses `--mamba-cache-mode align` (tests/evals/qwen4_exp/configs/Qwen3.8-Flash-Next-FP8-rocm.yaml).
- MooncakeStore (prefix-store semantics) excludes the ring group ([#55027](https://github.com/aarnphm/aarnphm.github.io/issues/55027)).

**P/D (NIXL)** — the “CSA-linear” path in `distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py`:

- **Detection.** CSA-linear is detected when any group is a `CircularBufferSpec` (:577-598). It allows at most one TP-replicated MambaSpec group (the PLE) with exactly one layer.
- **What transfers.**
  - The **ring IS transferred** (the scratch-region descriptors, :237-243). Unlike a prefix hit, the end of a P/D prompt is arbitrary, so the decode side needs the open group’s raw keys.
  - The **whole PLE conv page** is copied from one source rank, with descriptors sized by `_ple_block_len` and not by the region’s block_len ([#59997](https://github.com/aarnphm/aarnphm.github.io/issues/59997), :1912-1926, :1982).
- **Heterogeneous TP constraint (relevant to your hetero-TP work).** `_validate_csa_linear_tp_layout` (:967-989) rejects pairs that sit on opposite sides of the KV-head sharding boundary.
  - With 2 KV heads: P TP1 (2 heads per page) ↔ D TP≥2 (1 head per page, replicated) is **rejected**.
  - P TP2 ↔ D TP4 is fine.
  - Cause: the main KV and QSA side pages are packed in one block and move as whole block units.
- NIXL host staging is not supported with CSA-linear (:1397-1400). The decode side also needs the conv layout `VLLM_SSM_CONV_STATE_LAYOUT=DS` (:600-612).
- **`supports_kv_connector() = False` on `Qwen4ExpQSAFlashAttentionBackend`** (qsa.py:204-206). That flag only matters in backend _selection_ (`v1/attention/backend.py:339`), and the QSA owner hard-wires its backend (qsa.py:472). \[infer: it is effectively inert; P/D is exercised through the CSA-linear tests above.\]
- **PLE on the decode side** needs only `ngram_context` (the last 2 tokens), and model_state rebuilds it from the token history. Contrast DSV4.1 Engram, whose `hash_cache` slots don’t travel with KV and so needs `lookback_token_ids` (`vllm/models/deepseek_v41/common/engram.py:3-35`).

**CUDA graphs / DP / offload**

- **CUDA graphs.**
  - NVIDIA uses breakable CUDA graphs with no torch.compile ([#55272](https://github.com/aarnphm/aarnphm.github.io/issues/55272)).
  - Eager breaks sit at `_run_qsa` (qsa.py:553), `_short_conv` (ple_layer.py:334), `start_prefetch` and `_finalize_prefetch`.
  - QSA and the side-cache metadata builders are `UNIFORM_BATCH` (qsa.py:140-143, qsa_cache.py:599-602), so spec verify is treated as uniform decode.
- **Pinned prefetch.**
  - The buffer is allocated lazily but must exist before capture; capture raises otherwise (common/ngram_embedding.py:530-554).
  - The ids live in a persistent `_prefetch_ids` buffer because the side-stream lookup outlives the eager-break args, whose graph-pool memory a later segment can reuse ([#58489](https://github.com/aarnphm/aarnphm.github.io/issues/58489), ngram_embedding.py:225-230).
- **DP.**
  - With `embedding_across_dp`, every DP rank must run the PLE all_gather each step, including dummy batches, and needs `dp_metadata` (common/ngram_embedding.py:112-127).
  - `dp_shared_memory` is the default when DP > 1 and elastic EP is off.
- **Profiling.** The ring and compressed views must not hold extra storage refs, or the profiling KV cannot be freed ([#58961](https://github.com/aarnphm/aarnphm.github.io/issues/58961), qsa_cache.py:831-842).

---

## 4. PR reading order (vLLM main; `git show <sha>`)

| #  | PR                                                                                                                                                                | Title                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | Author           | Date          | Look at                                                                                                                                                                                                                                                                                                 |
| -- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 1  | [#53896](https://github.com/aarnphm/aarnphm.github.io/issues/53896) `e126687a9`                                                                                   | \[Model\] Support Qwen3.8-Flash-Next                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | Harry Huang      | 08-31         | 124 files. Read `qwen4_exp/{common,nvidia}` plus `kv_cache_interface.py` (CircularBufferSpec, MambaSpec.tp_replicated), `kv_cache_utils.py` (contiguous packing), NIXL `base_worker.py` (CSA-linear). The deleted `vllm/models/qwen4_exp/config.py` (git show) holds the config defaults and validators |
| 2  | [#54513](https://github.com/aarnphm/aarnphm.github.io/issues/54513) `003e34341`                                                                                   | Separate prefill and decode paths for QSA indexer                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | Thien Tran       | 09-02         | new `ops/qsa_indexer.py`; uniform-decode requirement; `qwen4_exp_qsa_warmup.py`                                                                                                                                                                                                                         |
| 3  | [#57097](https://github.com/aarnphm/aarnphm.github.io/issues/57097) `c9578f10a`                                                                                   | Fuse main QK-norm/RoPE/gate and KV write into the QSA pre-indexer                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | Shuolei Wang     | 10-01         | `ops/qsa_prepare.py`: everything one launch does; the `main_outputs` contract in indexer_qsa.py                                                                                                                                                                                                         |
| 4  | [#59533](https://github.com/aarnphm/aarnphm.github.io/issues/59533) `042ab0305`                                                                                   | Merge QSA QKVG and indexer QK projections                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | Thien Tran       | 10-05         | `Qwen4ExpQSAQKVIndexerLinear`: a replicated shard inside a TP-sharded merged linear (shard_rank tricks)                                                                                                                                                                                                 |
| 5  | [#54873](https://github.com/aarnphm/aarnphm.github.io/issues/54873) `31a8a2666`                                                                                   | Improve QSA sparse GQA for prefill/short-ctx decode                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | Thien Tran       | 09-04         | split-K selection and the trailing valid-count column                                                                                                                                                                                                                                                   |
| 6  | [#54915](https://github.com/aarnphm/aarnphm.github.io/issues/54915) `a5c9179e7` / [#57105](https://github.com/aarnphm/aarnphm.github.io/issues/57105) `7a877ae34` | Compact indexer logits workspace / avoid fragmentation                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | Thien Tran       | 09-04 / 09-27 | logits width = ⌈max_seq/c⌉ rounded to 64; worst-case alloc                                                                                                                                                                                                                                              |
| 7  | [#54890](https://github.com/aarnphm/aarnphm.github.io/issues/54890) `94e26dd3d`                                                                                   | FP8 indexer cache for QSA                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | Thien Tran       | 09-07         | scale-free e4m3 justified by RMSNorm                                                                                                                                                                                                                                                                    |
| 8  | [#55557](https://github.com/aarnphm/aarnphm.github.io/issues/55557) `dff1bde84`                                                                                   | fp8_e4m3 main KV on the QSA path                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | semerandre       | 09-16         | host-scale folding; bypassing the FA fp8 probe                                                                                                                                                                                                                                                          |
| 9  | [#54371](https://github.com/aarnphm/aarnphm.github.io/issues/54371) `3116c5d06`                                                                                   | UVA PLE-offload and Engram TP                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | Harry Huang      | 09-09         | `config/engram.py`, the ETP group in `parallel_state.py`, pinned UVA lookup                                                                                                                                                                                                                             |
| 10 | [#54517](https://github.com/aarnphm/aarnphm.github.io/issues/54517) `f870b9297`                                                                                   | Fuse Qwen4Exp PLE kernels                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | Thien Tran       | 09-02         | `ops/ple.py` gate/conv/ids math (docstring lines 3-11)                                                                                                                                                                                                                                                  |
| 11 | [#58957](https://github.com/aarnphm/aarnphm.github.io/issues/58957) `d6cce94fd`                                                                                   | Fuse HC down projection and SiLU on NVIDIA                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | Thien Tran       | 09-28         | CuTe-DSL skinny GEMM, M ≤ 48, warmup at capture sizes                                                                                                                                                                                                                                                   |
| 12 | [#54687](https://github.com/aarnphm/aarnphm.github.io/issues/54687) `fd4a15126`                                                                                   | Reuse HC combine-norm for MTP input                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | Canlin Guo       | 09-04         | how MTP injects the embedding as a pending block output                                                                                                                                                                                                                                                 |
| 13 | [#58489](https://github.com/aarnphm/aarnphm.github.io/issues/58489) `48d8880d0`                                                                                   | Keep pinned PLE prefetch ids out of the CUDA graph pool                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | Juntian Liu      | 09-24         | side-stream lifetime vs graph-pool reuse                                                                                                                                                                                                                                                                |
| 14 | [#59528](https://github.com/aarnphm/aarnphm.github.io/issues/59528) `240785b82`                                                                                   | Keep GLM-5.3 kpool tail and Qwen4 QSA ring out of the null block                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | Yifan Qiao       | 10-08         | `physical_block > 0` guard; dummy batches                                                                                                                                                                                                                                                               |
| 15 | [#59997](https://github.com/aarnphm/aarnphm.github.io/issues/59997) `72d59adc2`                                                                                   | NIXL: transfer the whole PLE short-conv page in P/D                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | Alberto Massidda | 10-08         | `_ple_block_len` vs region block_len in block-outer packing                                                                                                                                                                                                                                             |
| 16 | [#55027](https://github.com/aarnphm/aarnphm.github.io/issues/55027) `58d45fd76`                                                                                   | MooncakeStore: exclude non-prefix-cacheable (QSA ring) groups                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | Zhewen Li        | 09-13         | `prefix_cacheable` as a connector-visible property                                                                                                                                                                                                                                                      |
| 17 | [#57387](https://github.com/aarnphm/aarnphm.github.io/issues/57387) `e3cae8d2a`                                                                                   | Use upstream GLM-5.3 and Qwen4-Exp configs                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | Harry Mellor     | 10-02         | in-tree config deleted; layer-type rename `indexed_attention`                                                                                                                                                                                                                                           |
| 18 | [#58114](https://github.com/aarnphm/aarnphm.github.io/issues/58114) `20b52e9f5`                                                                                   | Reduce PLE metadata construction overhead                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | Jiangyun Zhu     | 09-28         | `short_conv_attn.py` PLE metadata path                                                                                                                                                                                                                                                                  |
| —  | also                                                                                                                                                              | [#55272](https://github.com/aarnphm/aarnphm.github.io/issues/55272) (no torch.compile on NVIDIA), [#58961](https://github.com/aarnphm/aarnphm.github.io/issues/58961) (profiling KV release), [#59945](https://github.com/aarnphm/aarnphm.github.io/issues/59945) (hea&#x64;_&#x64;im from unsharded heads), [#60023](https://github.com/aarnphm/aarnphm.github.io/issues/60023) (skip-layers), [#60142](https://github.com/aarnphm/aarnphm.github.io/issues/60142) (fp8 lookup below sm89), [#59443](https://github.com/aarnphm/aarnphm.github.io/issues/59443)/[#59990](https://github.com/aarnphm/aarnphm.github.io/issues/59990) (Quark/INC tables stay unquantized), [#57497](https://github.com/aarnphm/aarnphm.github.io/issues/57497)/[#60021](https://github.com/aarnphm/aarnphm.github.io/issues/60021) (ROCm PLE offload and fused PLE), [#50068](https://github.com/aarnphm/aarnphm.github.io/issues/50068) (Qwen3.8 = Qwen3.5-family on ROCm, \_not_ Flash-Next) |                  |               |                                                                                                                                                                                                                                                                                                         |

---

## 5. Sharp edges (bugs that expose invariants)

1. **The ring must never touch the null block ([#59528](https://github.com/aarnphm/aarnphm.github.io/issues/59528)).**
   - Dummy and padding requests sit on block 0. The ring formula `blk·cap + pos%cap` would scribble on the shared null page.
   - Guard: `valid &= physical_block > 0` (qsa_cache.py:130-132, 293-294). Every other spec type uses `>= 0`.
   - Invariant: per-request scratch (one block per request) may not alias the null block.
2. **Ring capacity under spec decode** (qsa_cache.py:845-855).
   - Draft rows are written before acceptance. Capacity is rounded up to a multiple of c and must divide block_size. Otherwise a rejected draft overwrites committed keys of the open group.
   - DSV4.1’s compressor sizes its ring as `max(8, pow2 ≥ num_spec+2)` instead (`models/deepseek_v41/compressor.py:150-153`).
3. **Profiling KV leak ([#58961](https://github.com/aarnphm/aarnphm.github.io/issues/58961)).** Caching `key_cache`/`rope_position_cache` views as attributes kept the profiling allocation alive after `clear_layer_kv_caches`. The fix derives the views on access (qsa_cache.py:831-842). Invariant: `kv_cache` must be the only strong reference to bound storage.
4. **Graph-pool aliasing ([#58489](https://github.com/aarnphm/aarnphm.github.io/issues/58489)).** Eager-break args are weak refs into the graph pool. The side-stream PLE lookup outlived them, and later graph segments reused the ids memory. The fix uses a persistent `_prefetch_ids`.
5. **head_dim with TP ([#59945](https://github.com/aarnphm/aarnphm.github.io/issues/59945)).** `hidden // num_heads` used the _sharded_ head count, giving the wrong head_dim at TP>1 when `config.head_dim` is unset (qsa.py:397).
6. **FP8 main KV vs the FlashAttention probe** (qsa.py:229-251). FlashAttentionImpl raises on fp8 KV where FA lacks support (sm120). QSA never runs FA, so the dtype is spoofed as “auto” during construction and restored afterwards.
7. **PLE pinned fp8 lookup below sm89 ([#60142](https://github.com/aarnphm/aarnphm.github.io/issues/60142)).** Triton cannot lower e4m3 there, so the lookup copies raw bytes (`uint8` pointers).
8. **Quantized-checkpoint tables.**
   - FP8 PLE must carry a global scale; the sentinel check is in `process_weights_after_loading` ([#54722](https://github.com/aarnphm/aarnphm.github.io/issues/54722), common/ngram_embedding.py:322-326).
   - Quark and INC configs must leave the tables BF16 ([#59443](https://github.com/aarnphm/aarnphm.github.io/issues/59443), [#59990](https://github.com/aarnphm/aarnphm.github.io/issues/59990)) (common/ngram_embedding.py:195-202).
9. **NIXL PLE page truncation ([#59997](https://github.com/aarnphm/aarnphm.github.io/issues/59997)).** In block-outer packing, the PLE page shares region 0 with a smaller main-KV page. Transfer sized by the region’s block_len silently dropped the tail of the conv state.
10. **Stale comment:** model.py:694 refers to `_maybe_add_ngram_kwargs`, which no longer exists. PLE inputs come from `Qwen4ExpModelState.prepare_inputs` (model_state.py:95-113).
11. **No torch.compile on NVIDIA** ([#55272](https://github.com/aarnphm/aarnphm.github.io/issues/55272)). The AMD path still lists `qwen4_exp_*` splitting ops (`config/compilation.py:774-778`). The two vendor trees diverge, and fixes often land twice (the `amd/` mirror).
12. **The QSA decode path assumes uniform decode query length.** It raises if the decode rows don’t form `num_decodes × q_len` (indexer_qsa.py:437-438). The metadata builder uses `split_decodes_and_prefills(require_uniform=True)` (qsa_cache.py:661-667).

---

