• ↑↓ pour naviguer
  • ↵ pour ouvrir
  • ⟶ pour sélectionner
  • ? pour demander
  • ⌘ ⌥ ↵ pour ouvrir dans un panneau
  • ←→ pour naviguer
  • esc pour rejeter
⌘ '
raccourcis clavier

source snapshot

main @ 2026-10-11. paths relative to repo root. M41 = vllm/models/deepseek_v41, M4 = vllm/models/deepseek_v4.

sources:

  • report: “DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression”, arXiv 2609.19969 (2026-09-17). read through a summarizer, not verbatim.
  • HF deepseek-ai/DeepSeek-V4.1-Flash/config.json and DeepSeek-V4-Flash/config.json, also read through a summarizer. Direct curl to huggingface.co is blocked by the proxy, so every config number below is second-hand. Cross-checks that hold: the 43-entry compress_ratios, 890 B/token, and the 183 GiB Engram size all reproduce from code arithmetic.
  • mHC = Manifold-Constrained Hyper-Connections (arXiv 2512.24880). Engram = “Conditional Memory via Scalable Lookup” (arXiv 2601.07372). Neither expansion appears in vLLM code; the code only says “hyper-connection”.
  • CSA = Compressed Sparse Attention (V4/V4.1 reports), CSA2 = its V4.1 version (report). HCA, V4’s ratio-128 layer, is “Heavily Compressed Attention” from memory of the V4 report; code only says “CSA/HCA” (M4/common/rope.py:32, M4/amd/rocm.py:686). not verified.
  • CED = Causal Encoder-Decoder (report). Not a vLLM term: in code it appears as kv_source_layer_ids plus “decoder replay”.

3. KV cache & state

3.1 per-layer spec registrations (NVIDIA, target model)

modulelayersspecblockper-state bytes
*.swa_cacheall 40 (+3 draft)SlidingWindowMLASpec(window=128, bounded_replay, model_version="deepseek_v4")32 tokens528/584
*.attn (compressed)2, 8, 14MLAAttentionSpec(tokens_per_state=2, state_content_bytes=288 or 528)cache_config.block_size288/528 per 2 tokens
*.attn (compressed)20same, tokens_per_state=1same288/528
*.indexer.k_cache2, 8, 14 (r2), 20 (r1)MLAAttentionSpec(head=68 or 132, uint8, block_stride_alignment=lcm(512, page_align) if sparse_logits)same68/132
*.compressor.state_cache2, 8, 14CircularBufferSpec(head=1024 fp32)capacity 8+4096 B/row

Refs: attention.py:1138-1163, 1206-1239; sparse_swa.py:116-140; compressor.py:159-167.

  • The packed compressed record keeps head_size=512 semantic, while state_content_bytes carries the real width (attention.py:1159-1162). alignment pads each page to the TMA stride: 512 on the V4.1 record, 576 on the V4 record.
  • Mixed page sizes are packed side by side inside each physical block. This needs a block-outermost layout (BLHNC); _get_packed_kv_cache_groups buckets layers into UniformTypeKVCacheSpecs groups that share one per-block layout (v1/core/kv_cache_utils.py:1860-1995), and an LB* layout errors with “set VLLM_KV_CACHE_LAYOUT=BLHNC” (:1575-1585). The indexer backends declare only BLHNC/BLNHC (v1/attention/backends/mla/indexer.py:314, 351). This ties directly into the BLHNC vs LBHNC RFC.
  • Circular-buffer buckets are “state buckets”: capped at the states a block already fits instead of widening the block (kv_cache_utils.py:1941-1972, changed in #56214).
  • Prefix-cache alignment: every prefix-cacheable spec with tokens_per_state>1 must divide the hit alignment (kv_cache_utils.py:887-897). Ratio 2 makes hits even, so the ratio-2 ring never holds an open group at a hit boundary. That is why the ring can be non-cacheable.
  • The drafter’s layers are found positionally: the group holding the last registered layer is flagged as the eagle group (_uses_trailing_mtp_layers / _annotate_eagle_groups, kv_cache_utils.py:1998-2060, “FIXME(yifan)”).
  • HiSparse rejects every spec with model_version == "deepseek_v4", V4.1 included (v1/hisparse/layout.py:100-104).

3.2 SWA bounded replay: encoder-side (#56227, scheduler)

  • Config: cache_config.swa_bounded_replay=True by default, excluded from the compile hash (config/cache.py:250-255, 314).
  • Forced off when not on MRv2, under PCP>1, or on ROCm (#57906) (attention.py:529-556). When off, the SWA group is prefix-cached like V4.
  • The scheduler reads one common prefix_replay_tokens window across groups (sched/scheduler.py:290-305). After a prefix hit of H tokens:
    1. Connector-reported hits are trimmed to a block boundary unless the load carries the window (:993-1004).
    2. A hit ≤ 128 is discarded entirely; recomputing it would save nothing (:1052-1066).
    3. replay_start = H − min(128, H), and num_computed_tokens is rewound by that amount (_mark_prefix_replay :3113-3131).
  • Worker (nvidia/model_state.py:338-367, _pad_replayed_slots_kernel :69-98): replayed tokens [replay_start, replay_start+128) get PAD_SLOT_ID in every prefix-cacheable group. The compressed slot mapping propagates PAD (v1/attention/backends/mla/compressor_utils.py:197-203). So replayed tokens rewrite only SWA KV; cached compressed and index K stay byte-identical.
  • The SWA window is lower-bounded by replay_start: visible_prefix = max(prefix_len − replay_start, 0) (sparse_swa.py:369-371, :1527). This is approximate: the replayed token’s SWA misses keys before s. Report: replay “truncates SWA to the replay segment”, accepts “approximate states”, and was simulated in post-training. Exact reconstruction would need L×128 tokens.
  • The persistent (SSD/host) footprint is about 1/8 of V4’s because the SWA groups are never stored (report).

3.3 decoder-side bounded replay (#58132, #59532, #59894)

  • cut = max(kv_source_layer_ids) = 20. In replay steps, layer 20 first calls write_kv: mHC pre, then attn.forward_kv, which writes SWA KV, compressed KV and index K for every row and skips wq_b, top-k and attention (nvidia/model.py:627-659, attention.py:689-715). Layers 20..39 then run on the kept rows (model.py:800-838, 1026-1035).
  • Kept rows per request: min(query_len, 128), or all rows when prompt_logprobs is set (model_state.py:269-276, 465-469). Decodes keep their single row.
  • Replay layers’ SWA lower bound: kv_start = max(replay_start, seq_len − 128) for trimmed requests (model_state.py:131-134).
  • Exception: layer 20’s own SWA metadata uses the batch’s replay_start, because write_kv covered every row (:529-542).
  • Before #59894 the cut was layer 21. DecoderReplayLayers then had to compact row_buffers (layer 20’s topk_indices_buffer / candidate_block_buffer) to replay rows. Now layer 20’s indexer runs inside the replay batch and publishes per-replay-row directly.
  • Runs: eager steps, and PIECEWISE graphs with num_tokens ≥ decoder_replay_trim_threshold (> window), which break out to a replay batch that can run its own PIECEWISE graph (decoder_replay_layers.py:70-130, nvidia/decoder_replay_cudagraph.py, #59532). FULL graphs and small PIECEWISE graphs keep the whole batch.
  • Under DP, all ranks agree, and an idle-rank dummy keeps 1 row (model_state.py:470-489).
  • Disabled (with warning) when (model.py:1202-1239):
    • PP splits layers ≥ cut;
    • SP, PCP or ubatching is on;
    • an Engram layer sits at or after the cut;
    • the drafter’s window exceeds 128 or the drafter is non-SWA, because DSpark reads aux hidden of 37–39, which exist only for kept rows.
  • This is the report’s CED “8B active in prefill vs 16B in decode”: O(N·L/2 + 128·L/2).

3.4 not ReplaySSM

ReplaySSM (cache_config.use_replayssm, replayssm_buffer_len=16; PRs #48018 by Johnny Liou, #52506 and #52928 by Andrii Skliar) is a Mamba2 decode kernel. It caches the last B SSM inputs and skips per-step full-state stores, writing the checkpoint on flush (config/cache.py:203-214). It reconstructs exact recurrent state for spec-decode rollback and prefix-cache alignment. SWA bounded replay instead recomputes tokens to rebuild a truncated attention window, is approximate, and is DSv4.1-only. The two share only the word “replay”.

3.5 P/D and offload

  • KVConnectorBase.get_loaded_kv_cache_group_ids defaults to the prefix-cacheable groups, i.e. what a hash-addressed store holds (kv_connector/v1/base.py:490-498). NIXL returns every transfer group, SWA included (nixl/connector.py:170-171).
  • The scheduler’s _load_restores_replay_window checks that set ⊇ the replay groups (sched/scheduler.py:3104-3111). If it holds: no replay, no block-trim, replay_start=0 (#59197).
  • Consequence: NIXL/Mooncake P/D loads take the window as is. Offload/store connectors (SimpleCPU, Mooncake store, tiering) hold no window, so D replays the last 128.
  • Prefiller backoff: with bounded replay, NIXL/Mooncake make P compute only N−1 prompt tokens, as for Mamba. The sliding window “is not replayed after a P/D load, so it must be laid out for the token the decoder recomputes” (nixl/base_scheduler.py:379-395, mooncake_connector.py:946-990).
  • SWA transfer groups are clipped to the last cdiv(128/bs)+1 blocks (mooncake_connector.py:916-944).
  • Engram’s slot hash_cache is not part of the KV cache, so a P/D load leaves it unwritten. MRv2 supplies lookback_token_ids from device token history, which take precedence (engram.py:27-35).
  • Heterogeneous TP: the compressed / index / SWA caches are all num_kv_heads=1 (max_tp_shards=1 on SWA), so they are replicated across TP ranks. Heterogeneous-TP P/D therefore moves whole-latent pages, with no head split (inferred from the specs; I did not trace NIXL’s descriptor code for V4.1).

See serving notes for model execution and PR history.

jump to heading

    kit de presse
    aperçu
    —
    format
    thème
    langue

    sélectionnez un passage pour le citer. *italique* **gras** `code` $tex$

    laisser vide pour utiliser le titre d’origine.

    inclure
    inclure
    blocs de la journée
    superposition
    graphiques
    ancre
    copier comme

    publier sur