Snapshot: vLLM main on 2026-10-11. Paths and PRs refer to that tree. Model-level architecture and code shape are in Qwen3.8-Next runtime notes.
Per-layer specs (TP=1, bf16; per-rank sizes shrink with TP except where marked replicated):
| Owner | Spec | Content | Size | Prefix-cacheable |
|---|---|---|---|---|
QSA main KV (layers.N.self_attn.attn) | FullAttentionSpec (qsa.py:543-551) | K,V of 2 heads × 256 | 2 KiB/token (1 KiB fp8) | yes |
QSA compressed index key (...indexer.compressed_key_cache) | MLAAttentionSpec(tokens_per_state=4) (qsa_cache.py:865-876) | RMSNorm+RoPE pooled key, 1 row per closed 4-token group, TP-replicated (ReplicatedLinear) | 256 B / 4 tokens = 64 B/token (32 B fp8) | yes; follows the main block table |
QSA raw-key ring (...indexer.raw_key_cache) | CircularBufferSpec (qsa_cache.py:844-862; class at v1/kv_cache_interface.py:909-947) | raw (pre-norm) keys of the open group, plus 3×int64 exact MRoPE positions packed after the key when MRoPE is used (qsa_cache.py:811-842) | capacity = c·⌈(c+num_spec)/c⌉ slots, 1 block per request for its whole life | no (prefix_cacheable=False) |
| GDN (36 layers) | MambaSpec (GDN_ATTN) | conv [10240, 3+num_spec] + SSM [48, 128, 128] | ≈1.5 MiB bf16 SSM + 80 KiB conv per layer [derived] | via the mamba cache mode (align / checkpoints) |
| PLE short conv (1 layer) | MambaSpec(SHORT_CONV, tp_replicated=True) (model.py:822-827) | conv history [10240, 9+num_spec] | 200 KiB @ num_spec=1, replicated on every TP rank | Mamba-style |
| PLE table | not KV; a weight | 51B params, pinned host | ≈51 GB fp8 / 102 GB bf16 per ETP group | — |
Cache invariants and the layout planner
- Ring sizing.
- It must hold the committed members of the open group plus every draft row written before acceptance. Too small, and a rejected draft overwrites a committed key that the next step still needs to close the group (qsa_cache.py:845-851; tests
test_qsa_circular_buffer_survives_one_speculative_step). - Its capacity must divide the attention block_size, because it joins the LCM that sets the scheduler block size (qsa_cache.py:852-855).
- Examples: num_spec=1 gives 8 slots; num_spec=5 gives 12 slots, which forces block_size to be a multiple of 12 [derived].
- It must hold the committed members of the open group plus every draft row written before acceptance. Too small, and a rejected draft overwrites a committed key that the next step still needs to close the group (qsa_cache.py:845-851; tests
- Ring slot mapping:
slot = block_table[req,0]·cap + pos % cap. Only the lastcaprows of each request’s chunk are written (token_idx + cap >= query_end). Requests on the null block (block 0) are skipped (qsa_cache.py:108-158, 281-297; #59528). - Compressed slot mapping: written only at group-closing positions
(pos+1)%c==0. It is masked by the main slot_mapping < 0 (qsa_cache.py:161-189, 298-320). - Packing.
QSAStateBackend.supported_kv_cache_layouts = (BLNHC, BLHNC). The reason given: “QSA pages are packed beside the main KV pages within each block” (qsa_cache.py:753-756).- Mixed page sizes force a block-compact layout, and with more than one group, a block-outermost one (
validate_kv_cache_layout,v1/core/kv_cache_utils.py:1555-1585,_get_packed_kv_cache_groups:1860+). - This is the BLHNC-vs-LBHNC layout RFC #42082 (
v1/kv_cache_layout.py:11-28) in practice. Qwen4Exp requires B-outer layouts.
- Mamba page size. Two Mamba specs (GDN, PLE) are present, and the platform takes the max page size (
vllm/platforms/interface.py:1044-1050).- The PLE page is TP-replicated while GDN shrinks with TP.
- [derived] At TP=8 with bf16 SSM: GDN ≈ 192 KiB + 15 KiB versus PLE 200–240 KiB. So PLE becomes the largest page at high TP or high num_spec.
Prefix caching
CircularBufferManagerclaims exactly 1 block per request. It reports a cache hit of 0, and its cache and skip hooks do nothing (v1/core/single_type_kv_cache_manager.py:1206-1295).- The coordinator excludes non-cacheable groups from hit lookup and from block-size divisibility (
v1/core/kv_cache_coordinator.py:670-679, 730-740). - Why this is correct [infer]: hits are aligned to block_size, which is a multiple of the ring capacity and so of c. At a hit boundary the open group is empty, so the compressor never needs ring rows from before the hit.
- GDN state needs mamba-cache-mode align or checkpoints. The ROCm eval config uses
--mamba-cache-mode align(tests/evals/qwen4_exp/configs/Qwen3.8-Flash-Next-FP8-rocm.yaml). - MooncakeStore (prefix-store semantics) excludes the ring group (#55027).
P/D (NIXL) — the “CSA-linear” path in distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py:
- Detection. CSA-linear is detected when any group is a
CircularBufferSpec(:577-598). It allows at most one TP-replicated MambaSpec group (the PLE) with exactly one layer. - What transfers.
- The ring IS transferred (the scratch-region descriptors, :237-243). Unlike a prefix hit, the end of a P/D prompt is arbitrary, so the decode side needs the open group’s raw keys.
- The whole PLE conv page is copied from one source rank, with descriptors sized by
_ple_block_lenand not by the region’s block_len (#59997, :1912-1926, :1982).
- Heterogeneous TP constraint (relevant to your hetero-TP work).
_validate_csa_linear_tp_layout(:967-989) rejects pairs that sit on opposite sides of the KV-head sharding boundary.- With 2 KV heads: P TP1 (2 heads per page) ↔ D TP≥2 (1 head per page, replicated) is rejected.
- P TP2 ↔ D TP4 is fine.
- Cause: the main KV and QSA side pages are packed in one block and move as whole block units.
- NIXL host staging is not supported with CSA-linear (:1397-1400). The decode side also needs the conv layout
VLLM_SSM_CONV_STATE_LAYOUT=DS(:600-612). supports_kv_connector() = FalseonQwen4ExpQSAFlashAttentionBackend(qsa.py:204-206). That flag only matters in backend selection (v1/attention/backend.py:339), and the QSA owner hard-wires its backend (qsa.py:472). [infer: it is effectively inert; P/D is exercised through the CSA-linear tests above.]- PLE on the decode side needs only
ngram_context(the last 2 tokens), and model_state rebuilds it from the token history. Contrast DSV4.1 Engram, whosehash_cacheslots don’t travel with KV and so needslookback_token_ids(vllm/models/deepseek_v41/common/engram.py:3-35).
CUDA graphs / DP / offload
- CUDA graphs.
- NVIDIA uses breakable CUDA graphs with no torch.compile (#55272).
- Eager breaks sit at
_run_qsa(qsa.py:553),_short_conv(ple_layer.py:334),start_prefetchand_finalize_prefetch. - QSA and the side-cache metadata builders are
UNIFORM_BATCH(qsa.py:140-143, qsa_cache.py:599-602), so spec verify is treated as uniform decode.
- Pinned prefetch.
- The buffer is allocated lazily but must exist before capture; capture raises otherwise (common/ngram_embedding.py:530-554).
- The ids live in a persistent
_prefetch_idsbuffer because the side-stream lookup outlives the eager-break args, whose graph-pool memory a later segment can reuse (#58489, ngram_embedding.py:225-230).
- DP.
- With
embedding_across_dp, every DP rank must run the PLE all_gather each step, including dummy batches, and needsdp_metadata(common/ngram_embedding.py:112-127). dp_shared_memoryis the default when DP > 1 and elastic EP is off.
- With
- Profiling. The ring and compressed views must not hold extra storage refs, or the profiling KV cannot be freed (#58961, qsa_cache.py:831-842).
| # | PR | Title | Author | Date | Look at |
|---|---|---|---|---|---|
| 1 | #53896 e126687a9 | [Model] Support Qwen3.8-Flash-Next | Harry Huang | 08-31 | 124 files. Read qwen4_exp/{common,nvidia} plus kv_cache_interface.py (CircularBufferSpec, MambaSpec.tp_replicated), kv_cache_utils.py (contiguous packing), NIXL base_worker.py (CSA-linear). The deleted vllm/models/qwen4_exp/config.py (git show) holds the config defaults and validators |
| 2 | #54513 003e34341 | Separate prefill and decode paths for QSA indexer | Thien Tran | 09-02 | new ops/qsa_indexer.py; uniform-decode requirement; qwen4_exp_qsa_warmup.py |
| 3 | #57097 c9578f10a | Fuse main QK-norm/RoPE/gate and KV write into the QSA pre-indexer | Shuolei Wang | 10-01 | ops/qsa_prepare.py: everything one launch does; the main_outputs contract in indexer_qsa.py |
| 4 | #59533 042ab0305 | Merge QSA QKVG and indexer QK projections | Thien Tran | 10-05 | Qwen4ExpQSAQKVIndexerLinear: a replicated shard inside a TP-sharded merged linear (shard_rank tricks) |
| 5 | #54873 31a8a2666 | Improve QSA sparse GQA for prefill/short-ctx decode | Thien Tran | 09-04 | split-K selection and the trailing valid-count column |
| 6 | #54915 a5c9179e7 / #57105 7a877ae34 | Compact indexer logits workspace / avoid fragmentation | Thien Tran | 09-04 / 09-27 | logits width = ⌈max_seq/c⌉ rounded to 64; worst-case alloc |
| 7 | #54890 94e26dd3d | FP8 indexer cache for QSA | Thien Tran | 09-07 | scale-free e4m3 justified by RMSNorm |
| 8 | #55557 dff1bde84 | fp8_e4m3 main KV on the QSA path | semerandre | 09-16 | host-scale folding; bypassing the FA fp8 probe |
| 9 | #54371 3116c5d06 | UVA PLE-offload and Engram TP | Harry Huang | 09-09 | config/engram.py, the ETP group in parallel_state.py, pinned UVA lookup |
| 10 | #54517 f870b9297 | Fuse Qwen4Exp PLE kernels | Thien Tran | 09-02 | ops/ple.py gate/conv/ids math (docstring lines 3-11) |
| 11 | #58957 d6cce94fd | Fuse HC down projection and SiLU on NVIDIA | Thien Tran | 09-28 | CuTe-DSL skinny GEMM, M ≤ 48, warmup at capture sizes |
| 12 | #54687 fd4a15126 | Reuse HC combine-norm for MTP input | Canlin Guo | 09-04 | how MTP injects the embedding as a pending block output |
| 13 | #58489 48d8880d0 | Keep pinned PLE prefetch ids out of the CUDA graph pool | Juntian Liu | 09-24 | side-stream lifetime vs graph-pool reuse |
| 14 | #59528 240785b82 | Keep GLM-5.3 kpool tail and Qwen4 QSA ring out of the null block | Yifan Qiao | 10-08 | physical_block > 0 guard; dummy batches |
| 15 | #59997 72d59adc2 | NIXL: transfer the whole PLE short-conv page in P/D | Alberto Massidda | 10-08 | _ple_block_len vs region block_len in block-outer packing |
| 16 | #55027 58d45fd76 | MooncakeStore: exclude non-prefix-cacheable (QSA ring) groups | Zhewen Li | 09-13 | prefix_cacheable as a connector-visible property |
| 17 | #57387 e3cae8d2a | Use upstream GLM-5.3 and Qwen4-Exp configs | Harry Mellor | 10-02 | in-tree config deleted; layer-type rename indexed_attention |
| 18 | #58114 20b52e9f5 | Reduce PLE metadata construction overhead | Jiangyun Zhu | 09-28 | short_conv_attn.py PLE metadata path |
| — | also | #55272 (no torch.compile on NVIDIA), #58961 (profiling KV release), #59945 (headdim from unsharded heads), #60023 (skip-layers), #60142 (fp8 lookup below sm89), #59443/#59990 (Quark/INC tables stay unquantized), #57497/#60021 (ROCm PLE offload and fused PLE), #50068 (Qwen3.8 = Qwen3.5-family on ROCm, _not Flash-Next) |
- The ring must never touch the null block (#59528).
- Dummy and padding requests sit on block 0. The ring formula
blk·cap + pos%capwould scribble on the shared null page. - Guard:
valid &= physical_block > 0(qsa_cache.py:130-132, 293-294). Every other spec type uses>= 0. - Invariant: per-request scratch (one block per request) may not alias the null block.
- Dummy and padding requests sit on block 0. The ring formula
- Ring capacity under spec decode (qsa_cache.py:845-855).
- Draft rows are written before acceptance. Capacity is rounded up to a multiple of c and must divide block_size. Otherwise a rejected draft overwrites committed keys of the open group.
- DSV4.1’s compressor sizes its ring as
max(8, pow2 ≥ num_spec+2)instead (models/deepseek_v41/compressor.py:150-153).
- Profiling KV leak (#58961). Caching
key_cache/rope_position_cacheviews as attributes kept the profiling allocation alive afterclear_layer_kv_caches. The fix derives the views on access (qsa_cache.py:831-842). Invariant:kv_cachemust be the only strong reference to bound storage. - Graph-pool aliasing (#58489). Eager-break args are weak refs into the graph pool. The side-stream PLE lookup outlived them, and later graph segments reused the ids memory. The fix uses a persistent
_prefetch_ids. - head_dim with TP (#59945).
hidden // num_headsused the sharded head count, giving the wrong head_dim at TP>1 whenconfig.head_dimis unset (qsa.py:397). - FP8 main KV vs the FlashAttention probe (qsa.py:229-251). FlashAttentionImpl raises on fp8 KV where FA lacks support (sm120). QSA never runs FA, so the dtype is spoofed as “auto” during construction and restored afterwards.
- PLE pinned fp8 lookup below sm89 (#60142). Triton cannot lower e4m3 there, so the lookup copies raw bytes (
uint8pointers). - Quantized-checkpoint tables.
- NIXL PLE page truncation (#59997). In block-outer packing, the PLE page shares region 0 with a smaller main-KV page. Transfer sized by the region’s block_len silently dropped the tail of the conv state.
- Stale comment: model.py:694 refers to
_maybe_add_ngram_kwargs, which no longer exists. PLE inputs come fromQwen4ExpModelState.prepare_inputs(model_state.py:95-113). - No torch.compile on NVIDIA (#55272). The AMD path still lists
qwen4_exp_*splitting ops (config/compilation.py:774-778). The two vendor trees diverge, and fixes often land twice (theamd/mirror). - The QSA decode path assumes uniform decode query length. It raises if the decode rows don’t form
num_decodes × q_len(indexer_qsa.py:437-438). The metadata builder usessplit_decodes_and_prefills(require_uniform=True)(qsa_cache.py:661-667).