• ↑↓ pour naviguer
  • ↵ pour ouvrir
  • ⟶ pour sélectionner
  • ? pour demander
  • ⌘ ⌥ ↵ pour ouvrir dans un panneau
  • ←→ pour naviguer
  • esc pour rejeter
⌘ '
raccourcis clavier

Qwen3.8-Next serving notes

Snapshot: vLLM main on 2026-10-11. Paths and PRs refer to that tree. Model-level architecture and code shape are in Qwen3.8-Next runtime notes.

3. KV cache & state (what lives where, per request)

Per-layer specs (TP=1, bf16; per-rank sizes shrink with TP except where marked replicated):

OwnerSpecContentSizePrefix-cacheable
QSA main KV (layers.N.self_attn.attn)FullAttentionSpec (qsa.py:543-551)K,V of 2 heads × 2562 KiB/token (1 KiB fp8)yes
QSA compressed index key (...indexer.compressed_key_cache)MLAAttentionSpec(tokens_per_state=4) (qsa_cache.py:865-876)RMSNorm+RoPE pooled key, 1 row per closed 4-token group, TP-replicated (ReplicatedLinear)256 B / 4 tokens = 64 B/token (32 B fp8)yes; follows the main block table
QSA raw-key ring (...indexer.raw_key_cache)CircularBufferSpec (qsa_cache.py:844-862; class at v1/kv_cache_interface.py:909-947)raw (pre-norm) keys of the open group, plus 3×int64 exact MRoPE positions packed after the key when MRoPE is used (qsa_cache.py:811-842)capacity = c·⌈(c+num_spec)/c⌉ slots, 1 block per request for its whole lifeno (prefix_cacheable=False)
GDN (36 layers)MambaSpec (GDN_ATTN)conv [10240, 3+num_spec] + SSM [48, 128, 128]≈1.5 MiB bf16 SSM + 80 KiB conv per layer [derived]via the mamba cache mode (align / checkpoints)
PLE short conv (1 layer)MambaSpec(SHORT_CONV, tp_replicated=True) (model.py:822-827)conv history [10240, 9+num_spec]200 KiB @ num_spec=1, replicated on every TP rankMamba-style
PLE tablenot KV; a weight51B params, pinned host≈51 GB fp8 / 102 GB bf16 per ETP group—

Cache invariants and the layout planner

  • Ring sizing.
    • It must hold the committed members of the open group plus every draft row written before acceptance. Too small, and a rejected draft overwrites a committed key that the next step still needs to close the group (qsa_cache.py:845-851; tests test_qsa_circular_buffer_survives_one_speculative_step).
    • Its capacity must divide the attention block_size, because it joins the LCM that sets the scheduler block size (qsa_cache.py:852-855).
    • Examples: num_spec=1 gives 8 slots; num_spec=5 gives 12 slots, which forces block_size to be a multiple of 12 [derived].
  • Ring slot mapping: slot = block_table[req,0]·cap + pos % cap. Only the last cap rows of each request’s chunk are written (token_idx + cap >= query_end). Requests on the null block (block 0) are skipped (qsa_cache.py:108-158, 281-297; #59528).
  • Compressed slot mapping: written only at group-closing positions (pos+1)%c==0. It is masked by the main slot_mapping < 0 (qsa_cache.py:161-189, 298-320).
  • Packing.
    • QSAStateBackend.supported_kv_cache_layouts = (BLNHC, BLHNC). The reason given: “QSA pages are packed beside the main KV pages within each block” (qsa_cache.py:753-756).
    • Mixed page sizes force a block-compact layout, and with more than one group, a block-outermost one (validate_kv_cache_layout, v1/core/kv_cache_utils.py:1555-1585, _get_packed_kv_cache_groups :1860+).
    • This is the BLHNC-vs-LBHNC layout RFC #42082 (v1/kv_cache_layout.py:11-28) in practice. Qwen4Exp requires B-outer layouts.
  • Mamba page size. Two Mamba specs (GDN, PLE) are present, and the platform takes the max page size (vllm/platforms/interface.py:1044-1050).
    • The PLE page is TP-replicated while GDN shrinks with TP.
    • [derived] At TP=8 with bf16 SSM: GDN ≈ 192 KiB + 15 KiB versus PLE 200–240 KiB. So PLE becomes the largest page at high TP or high num_spec.

Prefix caching

  • CircularBufferManager claims exactly 1 block per request. It reports a cache hit of 0, and its cache and skip hooks do nothing (v1/core/single_type_kv_cache_manager.py:1206-1295).
  • The coordinator excludes non-cacheable groups from hit lookup and from block-size divisibility (v1/core/kv_cache_coordinator.py:670-679, 730-740).
  • Why this is correct [infer]: hits are aligned to block_size, which is a multiple of the ring capacity and so of c. At a hit boundary the open group is empty, so the compressor never needs ring rows from before the hit.
  • GDN state needs mamba-cache-mode align or checkpoints. The ROCm eval config uses --mamba-cache-mode align (tests/evals/qwen4_exp/configs/Qwen3.8-Flash-Next-FP8-rocm.yaml).
  • MooncakeStore (prefix-store semantics) excludes the ring group (#55027).

P/D (NIXL) — the “CSA-linear” path in distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py:

  • Detection. CSA-linear is detected when any group is a CircularBufferSpec (:577-598). It allows at most one TP-replicated MambaSpec group (the PLE) with exactly one layer.
  • What transfers.
    • The ring IS transferred (the scratch-region descriptors, :237-243). Unlike a prefix hit, the end of a P/D prompt is arbitrary, so the decode side needs the open group’s raw keys.
    • The whole PLE conv page is copied from one source rank, with descriptors sized by _ple_block_len and not by the region’s block_len (#59997, :1912-1926, :1982).
  • Heterogeneous TP constraint (relevant to your hetero-TP work). _validate_csa_linear_tp_layout (:967-989) rejects pairs that sit on opposite sides of the KV-head sharding boundary.
    • With 2 KV heads: P TP1 (2 heads per page) ↔ D TP≥2 (1 head per page, replicated) is rejected.
    • P TP2 ↔ D TP4 is fine.
    • Cause: the main KV and QSA side pages are packed in one block and move as whole block units.
  • NIXL host staging is not supported with CSA-linear (:1397-1400). The decode side also needs the conv layout VLLM_SSM_CONV_STATE_LAYOUT=DS (:600-612).
  • supports_kv_connector() = False on Qwen4ExpQSAFlashAttentionBackend (qsa.py:204-206). That flag only matters in backend selection (v1/attention/backend.py:339), and the QSA owner hard-wires its backend (qsa.py:472). [infer: it is effectively inert; P/D is exercised through the CSA-linear tests above.]
  • PLE on the decode side needs only ngram_context (the last 2 tokens), and model_state rebuilds it from the token history. Contrast DSV4.1 Engram, whose hash_cache slots don’t travel with KV and so needs lookback_token_ids (vllm/models/deepseek_v41/common/engram.py:3-35).

CUDA graphs / DP / offload

  • CUDA graphs.
    • NVIDIA uses breakable CUDA graphs with no torch.compile (#55272).
    • Eager breaks sit at _run_qsa (qsa.py:553), _short_conv (ple_layer.py:334), start_prefetch and _finalize_prefetch.
    • QSA and the side-cache metadata builders are UNIFORM_BATCH (qsa.py:140-143, qsa_cache.py:599-602), so spec verify is treated as uniform decode.
  • Pinned prefetch.
    • The buffer is allocated lazily but must exist before capture; capture raises otherwise (common/ngram_embedding.py:530-554).
    • The ids live in a persistent _prefetch_ids buffer because the side-stream lookup outlives the eager-break args, whose graph-pool memory a later segment can reuse (#58489, ngram_embedding.py:225-230).
  • DP.
    • With embedding_across_dp, every DP rank must run the PLE all_gather each step, including dummy batches, and needs dp_metadata (common/ngram_embedding.py:112-127).
    • dp_shared_memory is the default when DP > 1 and elastic EP is off.
  • Profiling. The ring and compressed views must not hold extra storage refs, or the profiling KV cannot be freed (#58961, qsa_cache.py:831-842).

4. PR reading order (vLLM main; git show <sha>)

#PRTitleAuthorDateLook at
1#53896 e126687a9[Model] Support Qwen3.8-Flash-NextHarry Huang08-31124 files. Read qwen4_exp/{common,nvidia} plus kv_cache_interface.py (CircularBufferSpec, MambaSpec.tp_replicated), kv_cache_utils.py (contiguous packing), NIXL base_worker.py (CSA-linear). The deleted vllm/models/qwen4_exp/config.py (git show) holds the config defaults and validators
2#54513 003e34341Separate prefill and decode paths for QSA indexerThien Tran09-02new ops/qsa_indexer.py; uniform-decode requirement; qwen4_exp_qsa_warmup.py
3#57097 c9578f10aFuse main QK-norm/RoPE/gate and KV write into the QSA pre-indexerShuolei Wang10-01ops/qsa_prepare.py: everything one launch does; the main_outputs contract in indexer_qsa.py
4#59533 042ab0305Merge QSA QKVG and indexer QK projectionsThien Tran10-05Qwen4ExpQSAQKVIndexerLinear: a replicated shard inside a TP-sharded merged linear (shard_rank tricks)
5#54873 31a8a2666Improve QSA sparse GQA for prefill/short-ctx decodeThien Tran09-04split-K selection and the trailing valid-count column
6#54915 a5c9179e7 / #57105 7a877ae34Compact indexer logits workspace / avoid fragmentationThien Tran09-04 / 09-27logits width = ⌈max_seq/c⌉ rounded to 64; worst-case alloc
7#54890 94e26dd3dFP8 indexer cache for QSAThien Tran09-07scale-free e4m3 justified by RMSNorm
8#55557 dff1bde84fp8_e4m3 main KV on the QSA pathsemerandre09-16host-scale folding; bypassing the FA fp8 probe
9#54371 3116c5d06UVA PLE-offload and Engram TPHarry Huang09-09config/engram.py, the ETP group in parallel_state.py, pinned UVA lookup
10#54517 f870b9297Fuse Qwen4Exp PLE kernelsThien Tran09-02ops/ple.py gate/conv/ids math (docstring lines 3-11)
11#58957 d6cce94fdFuse HC down projection and SiLU on NVIDIAThien Tran09-28CuTe-DSL skinny GEMM, M ≤ 48, warmup at capture sizes
12#54687 fd4a15126Reuse HC combine-norm for MTP inputCanlin Guo09-04how MTP injects the embedding as a pending block output
13#58489 48d8880d0Keep pinned PLE prefetch ids out of the CUDA graph poolJuntian Liu09-24side-stream lifetime vs graph-pool reuse
14#59528 240785b82Keep GLM-5.3 kpool tail and Qwen4 QSA ring out of the null blockYifan Qiao10-08physical_block > 0 guard; dummy batches
15#59997 72d59adc2NIXL: transfer the whole PLE short-conv page in P/DAlberto Massidda10-08_ple_block_len vs region block_len in block-outer packing
16#55027 58d45fd76MooncakeStore: exclude non-prefix-cacheable (QSA ring) groupsZhewen Li09-13prefix_cacheable as a connector-visible property
17#57387 e3cae8d2aUse upstream GLM-5.3 and Qwen4-Exp configsHarry Mellor10-02in-tree config deleted; layer-type rename indexed_attention
18#58114 20b52e9f5Reduce PLE metadata construction overheadJiangyun Zhu09-28short_conv_attn.py PLE metadata path
—also#55272 (no torch.compile on NVIDIA), #58961 (profiling KV release), #59945 (headdim from unsharded heads), #60023 (skip-layers), #60142 (fp8 lookup below sm89), #59443/#59990 (Quark/INC tables stay unquantized), #57497/#60021 (ROCm PLE offload and fused PLE), #50068 (Qwen3.8 = Qwen3.5-family on ROCm, _not Flash-Next)

5. Sharp edges (bugs that expose invariants)

  1. The ring must never touch the null block (#59528).
    • Dummy and padding requests sit on block 0. The ring formula blk·cap + pos%cap would scribble on the shared null page.
    • Guard: valid &= physical_block > 0 (qsa_cache.py:130-132, 293-294). Every other spec type uses >= 0.
    • Invariant: per-request scratch (one block per request) may not alias the null block.
  2. Ring capacity under spec decode (qsa_cache.py:845-855).
    • Draft rows are written before acceptance. Capacity is rounded up to a multiple of c and must divide block_size. Otherwise a rejected draft overwrites committed keys of the open group.
    • DSV4.1’s compressor sizes its ring as max(8, pow2 ≥ num_spec+2) instead (models/deepseek_v41/compressor.py:150-153).
  3. Profiling KV leak (#58961). Caching key_cache/rope_position_cache views as attributes kept the profiling allocation alive after clear_layer_kv_caches. The fix derives the views on access (qsa_cache.py:831-842). Invariant: kv_cache must be the only strong reference to bound storage.
  4. Graph-pool aliasing (#58489). Eager-break args are weak refs into the graph pool. The side-stream PLE lookup outlived them, and later graph segments reused the ids memory. The fix uses a persistent _prefetch_ids.
  5. head_dim with TP (#59945). hidden // num_heads used the sharded head count, giving the wrong head_dim at TP>1 when config.head_dim is unset (qsa.py:397).
  6. FP8 main KV vs the FlashAttention probe (qsa.py:229-251). FlashAttentionImpl raises on fp8 KV where FA lacks support (sm120). QSA never runs FA, so the dtype is spoofed as “auto” during construction and restored afterwards.
  7. PLE pinned fp8 lookup below sm89 (#60142). Triton cannot lower e4m3 there, so the lookup copies raw bytes (uint8 pointers).
  8. Quantized-checkpoint tables.
    • FP8 PLE must carry a global scale; the sentinel check is in process_weights_after_loading (#54722, common/ngram_embedding.py:322-326).
    • Quark and INC configs must leave the tables BF16 (#59443, #59990) (common/ngram_embedding.py:195-202).
  9. NIXL PLE page truncation (#59997). In block-outer packing, the PLE page shares region 0 with a smaller main-KV page. Transfer sized by the region’s block_len silently dropped the tail of the conv state.
  10. Stale comment: model.py:694 refers to _maybe_add_ngram_kwargs, which no longer exists. PLE inputs come from Qwen4ExpModelState.prepare_inputs (model_state.py:95-113).
  11. No torch.compile on NVIDIA (#55272). The AMD path still lists qwen4_exp_* splitting ops (config/compilation.py:774-778). The two vendor trees diverge, and fixes often land twice (the amd/ mirror).
  12. The QSA decode path assumes uniform decode query length. It raises if the decode rows don’t form num_decodes × q_len (indexer_qsa.py:437-438). The metadata builder uses split_decodes_and_prefills(require_uniform=True) (qsa_cache.py:661-667).
jump to heading

    kit de presse
    aperçu
    —
    format
    thème
    langue

    sélectionnez un passage pour le citer. *italique* **gras** `code` $tex$

    laisser vide pour utiliser le titre d’origine.

    inclure
    inclure
    blocs de la journée
    superposition
    graphiques
    ancre
    copier comme

    publier sur