main @ 2026-10-11. paths relative to repo root. M41 = vllm/models/deepseek_v41, M4 = vllm/models/deepseek_v4.
sources:
- report: “DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression”, arXiv 2609.19969 (2026-09-17). read through a summarizer, not verbatim.
- HF
deepseek-ai/DeepSeek-V4.1-Flash/config.jsonandDeepSeek-V4-Flash/config.json, also read through a summarizer. Direct curl to huggingface.co is blocked by the proxy, so every config number below is second-hand. Cross-checks that hold: the 43-entrycompress_ratios, 890 B/token, and the 183 GiB Engram size all reproduce from code arithmetic. - mHC = Manifold-Constrained Hyper-Connections (arXiv 2512.24880). Engram = “Conditional Memory via Scalable Lookup” (arXiv 2601.07372). Neither expansion appears in vLLM code; the code only says “hyper-connection”.
- CSA = Compressed Sparse Attention (V4/V4.1 reports), CSA2 = its V4.1 version (report). HCA, V4’s ratio-128 layer, is “Heavily Compressed Attention” from memory of the V4 report; code only says “CSA/HCA” (
M4/common/rope.py:32,M4/amd/rocm.py:686). not verified. - CED = Causal Encoder-Decoder (report). Not a vLLM term: in code it appears as
kv_source_layer_idsplus “decoder replay”.
| module | layers | spec | block | per-state bytes |
|---|---|---|---|---|
*.swa_cache | all 40 (+3 draft) | SlidingWindowMLASpec(window=128, bounded_replay, model_version="deepseek_v4") | 32 tokens | 528/584 |
*.attn (compressed) | 2, 8, 14 | MLAAttentionSpec(tokens_per_state=2, state_content_bytes=288 or 528) | cache_config.block_size | 288/528 per 2 tokens |
*.attn (compressed) | 20 | same, tokens_per_state=1 | same | 288/528 |
*.indexer.k_cache | 2, 8, 14 (r2), 20 (r1) | MLAAttentionSpec(head=68 or 132, uint8, block_stride_alignment=lcm(512, page_align) if sparse_logits) | same | 68/132 |
*.compressor.state_cache | 2, 8, 14 | CircularBufferSpec(head=1024 fp32) | capacity 8+ | 4096 B/row |
Refs: attention.py:1138-1163, 1206-1239; sparse_swa.py:116-140; compressor.py:159-167.
- The packed compressed record keeps
head_size=512semantic, whilestate_content_bytescarries the real width (attention.py:1159-1162).alignmentpads each page to the TMA stride: 512 on the V4.1 record, 576 on the V4 record. - Mixed page sizes are packed side by side inside each physical block. This needs a block-outermost layout (BLHNC);
_get_packed_kv_cache_groupsbuckets layers intoUniformTypeKVCacheSpecsgroups that share one per-block layout (v1/core/kv_cache_utils.py:1860-1995), and an LB* layout errors with “set VLLM_KV_CACHE_LAYOUT=BLHNC” (:1575-1585). The indexer backends declare only BLHNC/BLNHC (v1/attention/backends/mla/indexer.py:314, 351). This ties directly into the BLHNC vs LBHNC RFC. - Circular-buffer buckets are “state buckets”: capped at the states a block already fits instead of widening the block (
kv_cache_utils.py:1941-1972, changed in #56214). - Prefix-cache alignment: every prefix-cacheable spec with
tokens_per_state>1must divide the hit alignment (kv_cache_utils.py:887-897). Ratio 2 makes hits even, so the ratio-2 ring never holds an open group at a hit boundary. That is why the ring can be non-cacheable. - The drafter’s layers are found positionally: the group holding the last registered layer is flagged as the eagle group (
_uses_trailing_mtp_layers/_annotate_eagle_groups,kv_cache_utils.py:1998-2060, “FIXME(yifan)”). - HiSparse rejects every spec with
model_version == "deepseek_v4", V4.1 included (v1/hisparse/layout.py:100-104).
- Config:
cache_config.swa_bounded_replay=Trueby default, excluded from the compile hash (config/cache.py:250-255, 314). - Forced off when not on MRv2, under PCP>1, or on ROCm (#57906) (
attention.py:529-556). When off, the SWA group is prefix-cached like V4. - The scheduler reads one common
prefix_replay_tokenswindow across groups (sched/scheduler.py:290-305). After a prefix hit of H tokens:- Connector-reported hits are trimmed to a block boundary unless the load carries the window (
:993-1004). - A hit ≤ 128 is discarded entirely; recomputing it would save nothing (
:1052-1066). replay_start = H − min(128, H), andnum_computed_tokensis rewound by that amount (_mark_prefix_replay :3113-3131).
- Connector-reported hits are trimmed to a block boundary unless the load carries the window (
- Worker (
nvidia/model_state.py:338-367,_pad_replayed_slots_kernel :69-98): replayed tokens[replay_start, replay_start+128)getPAD_SLOT_IDin every prefix-cacheable group. The compressed slot mapping propagates PAD (v1/attention/backends/mla/compressor_utils.py:197-203). So replayed tokens rewrite only SWA KV; cached compressed and index K stay byte-identical. - The SWA window is lower-bounded by
replay_start:visible_prefix = max(prefix_len − replay_start, 0)(sparse_swa.py:369-371,:1527). This is approximate: the replayed token’s SWA misses keys befores. Report: replay “truncates SWA to the replay segment”, accepts “approximate states”, and was simulated in post-training. Exact reconstruction would need L×128 tokens. - The persistent (SSD/host) footprint is about 1/8 of V4’s because the SWA groups are never stored (report).
cut = max(kv_source_layer_ids) = 20. In replay steps, layer 20 first callswrite_kv: mHC pre, thenattn.forward_kv, which writes SWA KV, compressed KV and index K for every row and skips wq_b, top-k and attention (nvidia/model.py:627-659,attention.py:689-715). Layers 20..39 then run on the kept rows (model.py:800-838, 1026-1035).- Kept rows per request:
min(query_len, 128), or all rows whenprompt_logprobsis set (model_state.py:269-276, 465-469). Decodes keep their single row. - Replay layers’ SWA lower bound:
kv_start = max(replay_start, seq_len − 128)for trimmed requests (model_state.py:131-134). - Exception: layer 20’s own SWA metadata uses the batch’s
replay_start, because write_kv covered every row (:529-542). - Before #59894 the cut was layer 21.
DecoderReplayLayersthen had to compactrow_buffers(layer 20’stopk_indices_buffer/candidate_block_buffer) to replay rows. Now layer 20’s indexer runs inside the replay batch and publishes per-replay-row directly. - Runs: eager steps, and PIECEWISE graphs with
num_tokens ≥ decoder_replay_trim_threshold(> window), which break out to a replay batch that can run its own PIECEWISE graph (decoder_replay_layers.py:70-130,nvidia/decoder_replay_cudagraph.py, #59532). FULL graphs and small PIECEWISE graphs keep the whole batch. - Under DP, all ranks agree, and an idle-rank dummy keeps 1 row (
model_state.py:470-489). - Disabled (with warning) when (
model.py:1202-1239):- PP splits layers ≥ cut;
- SP, PCP or ubatching is on;
- an Engram layer sits at or after the cut;
- the drafter’s window exceeds 128 or the drafter is non-SWA, because DSpark reads aux hidden of 37–39, which exist only for kept rows.
- This is the report’s CED “8B active in prefill vs 16B in decode”: O(N·L/2 + 128·L/2).
ReplaySSM (cache_config.use_replayssm, replayssm_buffer_len=16; PRs #48018 by Johnny Liou, #52506 and #52928 by Andrii Skliar) is a Mamba2 decode kernel. It caches the last B SSM inputs and skips per-step full-state stores, writing the checkpoint on flush (config/cache.py:203-214). It reconstructs exact recurrent state for spec-decode rollback and prefix-cache alignment. SWA bounded replay instead recomputes tokens to rebuild a truncated attention window, is approximate, and is DSv4.1-only. The two share only the word “replay”.
KVConnectorBase.get_loaded_kv_cache_group_idsdefaults to the prefix-cacheable groups, i.e. what a hash-addressed store holds (kv_connector/v1/base.py:490-498). NIXL returns every transfer group, SWA included (nixl/connector.py:170-171).- The scheduler’s
_load_restores_replay_windowchecks that set ⊇ the replay groups (sched/scheduler.py:3104-3111). If it holds: no replay, no block-trim,replay_start=0(#59197). - Consequence: NIXL/Mooncake P/D loads take the window as is. Offload/store connectors (SimpleCPU, Mooncake store, tiering) hold no window, so D replays the last 128.
- Prefiller backoff: with bounded replay, NIXL/Mooncake make P compute only N−1 prompt tokens, as for Mamba. The sliding window “is not replayed after a P/D load, so it must be laid out for the token the decoder recomputes” (
nixl/base_scheduler.py:379-395,mooncake_connector.py:946-990). - SWA transfer groups are clipped to the last
cdiv(128/bs)+1blocks (mooncake_connector.py:916-944). - Engram’s slot
hash_cacheis not part of the KV cache, so a P/D load leaves it unwritten. MRv2 supplieslookback_token_idsfrom device token history, which take precedence (engram.py:27-35). - Heterogeneous TP: the compressed / index / SWA caches are all
num_kv_heads=1(max_tp_shards=1on SWA), so they are replicated across TP ranks. Heterogeneous-TP P/D therefore moves whole-latent pages, with no head split (inferred from the specs; I did not trace NIXL’s descriptor code for V4.1).
See serving notes for model execution and PR history.