---
title: DeepSeek V4.1 cache and state
tags:
  - inference
  - vllm
pageLayout: default
slug: thoughts/inference/deepseek-v4.1-state
permalink: https://aarnphm.xyz/thoughts/inference/deepseek-v4.1-state.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
## source snapshot

main @ 2026-10-11. paths relative to repo root. `M41` = `vllm/models/deepseek_v41`, `M4` = `vllm/models/deepseek_v4`.

sources:

- report: “DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression”, arXiv 2609.19969 (2026-09-17). read through a summarizer, not verbatim.
- HF `deepseek-ai/DeepSeek-V4.1-Flash/config.json` and `DeepSeek-V4-Flash/config.json`, also read through a summarizer. Direct curl to huggingface.co is blocked by the proxy, so every config number below is second-hand. Cross-checks that hold: the 43-entry `compress_ratios`, 890 B/token, and the 183 GiB Engram size all reproduce from code arithmetic.
- mHC = Manifold-Constrained Hyper-Connections (arXiv 2512.24880). Engram = “Conditional Memory via Scalable Lookup” (arXiv 2601.07372). Neither expansion appears in vLLM code; the code only says “hyper-connection”.
- CSA = Compressed Sparse Attention (V4/V4.1 reports), CSA2 = its V4.1 version (report). HCA, V4’s ratio-128 layer, is “Heavily Compressed Attention” from memory of the V4 report; code only says “CSA/HCA” (`M4/common/rope.py:32`, `M4/amd/rocm.py:686`). **not verified.**
- CED = Causal Encoder-Decoder (report). Not a vLLM term: in code it appears as `kv_source_layer_ids` plus “decoder replay”.

## 3. KV cache & state

### 3.1 per-layer spec registrations (NVIDIA, target model)

| module                     | layers                 | spec                                                                                                    | block                     | per-state bytes      |
| -------------------------- | ---------------------- | ------------------------------------------------------------------------------------------------------- | ------------------------- | -------------------- |
| `*.swa_cache`              | all 40 (+3 draft)      | `SlidingWindowMLASpec(window=128, bounded_replay, model_version="deepseek_v4")`                         | 32 tokens                 | 528/584              |
| `*.attn` (compressed)      | 2, 8, 14               | `MLAAttentionSpec(tokens_per_state=2, state_content_bytes=288 or 528)`                                  | `cache_config.block_size` | 288/528 per 2 tokens |
| `*.attn` (compressed)      | 20                     | same, `tokens_per_state=1`                                                                              | same                      | 288/528              |
| `*.indexer.k_cache`        | 2, 8, 14 (r2), 20 (r1) | `MLAAttentionSpec(head=68 or 132, uint8, block_stride_alignment=lcm(512, page_align) if sparse_logits)` | same                      | 68/132               |
| `*.compressor.state_cache` | 2, 8, 14               | `CircularBufferSpec(head=1024 fp32)`                                                                    | capacity 8+               | 4096 B/row           |

Refs: `attention.py:1138-1163, 1206-1239`; `sparse_swa.py:116-140`; `compressor.py:159-167`.

- The packed compressed record keeps `head_size=512` semantic, while `state_content_bytes` carries the real width (`attention.py:1159-1162`). `alignment` pads each page to the TMA stride: 512 on the V4.1 record, 576 on the V4 record.
- Mixed page sizes are packed side by side inside each physical block. This needs a **block-outermost layout (BLHNC)**; `_get_packed_kv_cache_groups` buckets layers into `UniformTypeKVCacheSpecs` groups that share one per-block layout (`v1/core/kv_cache_utils.py:1860-1995`), and an LB\* layout errors with “set VLLM_KV_CACHE_LAYOUT=BLHNC” (`:1575-1585`). The indexer backends declare only BLHNC/BLNHC (`v1/attention/backends/mla/indexer.py:314, 351`). This ties directly into the BLHNC vs LBHNC RFC.
- Circular-buffer buckets are “state buckets”: capped at the states a block already fits instead of widening the block (`kv_cache_utils.py:1941-1972`, changed in [#56214](https://github.com/aarnphm/aarnphm.github.io/issues/56214)).
- Prefix-cache alignment: every prefix-cacheable spec with `tokens_per_state>1` must divide the hit alignment (`kv_cache_utils.py:887-897`). Ratio 2 makes hits even, so the ratio-2 ring never holds an open group at a hit boundary. That is why the ring can be non-cacheable.
- The drafter’s layers are found **positionally**: the group holding the last registered layer is flagged as the eagle group (`_uses_trailing_mtp_layers` / `_annotate_eagle_groups`, `kv_cache_utils.py:1998-2060`, “FIXME(yifan)”).
- HiSparse rejects every spec with `model_version == "deepseek_v4"`, V4.1 included (`v1/hisparse/layout.py:100-104`).

### 3.2 SWA bounded replay: encoder-side ([#56227](https://github.com/aarnphm/aarnphm.github.io/issues/56227), scheduler)

- Config: `cache_config.swa_bounded_replay=True` by default, excluded from the compile hash (`config/cache.py:250-255, 314`).
- Forced off when not on MRv2, under PCP>1, or on ROCm ([#57906](https://github.com/aarnphm/aarnphm.github.io/issues/57906)) (`attention.py:529-556`). When off, the SWA group is prefix-cached like V4.
- The scheduler reads one common `prefix_replay_tokens` window across groups (`sched/scheduler.py:290-305`). After a prefix hit of H tokens:
  1. Connector-reported hits are trimmed to a block boundary unless the load carries the window (`:993-1004`).
  2. A hit ≤ 128 is discarded entirely; recomputing it would save nothing (`:1052-1066`).
  3. `replay_start = H − min(128, H)`, and `num_computed_tokens` is rewound by that amount (`_mark_prefix_replay :3113-3131`).
- Worker (`nvidia/model_state.py:338-367`, `_pad_replayed_slots_kernel :69-98`): replayed tokens `[replay_start, replay_start+128)` get `PAD_SLOT_ID` in every **prefix-cacheable** group. The compressed slot mapping propagates PAD (`v1/attention/backends/mla/compressor_utils.py:197-203`). So replayed tokens rewrite only SWA KV; cached compressed and index K stay byte-identical.
- The SWA window is lower-bounded by `replay_start`: `visible_prefix = max(prefix_len − replay_start, 0)` (`sparse_swa.py:369-371`, `:1527`). This is **approximate**: the replayed token’s SWA misses keys before `s`. Report: replay “truncates SWA to the replay segment”, accepts “approximate states”, and was simulated in post-training. Exact reconstruction would need L×128 tokens.
- The persistent (SSD/host) footprint is about 1/8 of V4’s because the SWA groups are never stored (report).

### 3.3 decoder-side bounded replay ([#58132](https://github.com/aarnphm/aarnphm.github.io/issues/58132), [#59532](https://github.com/aarnphm/aarnphm.github.io/issues/59532), [#59894](https://github.com/aarnphm/aarnphm.github.io/issues/59894))

- `cut = max(kv_source_layer_ids) = 20`. In replay steps, layer 20 first calls `write_kv`: mHC pre, then `attn.forward_kv`, which writes SWA KV, compressed KV and index K for **every** row and skips wq_b, top-k and attention (`nvidia/model.py:627-659`, `attention.py:689-715`). Layers 20..39 then run on the kept rows (`model.py:800-838, 1026-1035`).
- Kept rows per request: `min(query_len, 128)`, or all rows when `prompt_logprobs` is set (`model_state.py:269-276, 465-469`). Decodes keep their single row.
- Replay layers’ SWA lower bound: `kv_start = max(replay_start, seq_len − 128)` for trimmed requests (`model_state.py:131-134`).
- Exception: layer 20’s own SWA metadata uses the batch’s `replay_start`, because write_kv covered every row (`:529-542`).
- Before [#59894](https://github.com/aarnphm/aarnphm.github.io/issues/59894) the cut was layer 21. `DecoderReplayLayers` then had to compact `row_buffers` (layer 20’s `topk_indices_buffer` / `candidate_block_buffer`) to replay rows. Now layer 20’s indexer runs inside the replay batch and publishes per-replay-row directly.
- Runs: eager steps, and PIECEWISE graphs with `num_tokens ≥ decoder_replay_trim_threshold` (> window), which break out to a replay batch that can run its own PIECEWISE graph (`decoder_replay_layers.py:70-130`, `nvidia/decoder_replay_cudagraph.py`, [#59532](https://github.com/aarnphm/aarnphm.github.io/issues/59532)). FULL graphs and small PIECEWISE graphs keep the whole batch.
- Under DP, all ranks agree, and an idle-rank dummy keeps 1 row (`model_state.py:470-489`).
- Disabled (with warning) when (`model.py:1202-1239`):
  - PP splits layers ≥ cut;
  - SP, PCP or ubatching is on;
  - an Engram layer sits at or after the cut;
  - the drafter’s window exceeds 128 or the drafter is non-SWA, because DSpark reads aux hidden of 37–39, which exist only for kept rows.
- This is the report’s CED “8B active in prefill vs 16B in decode”: O(N·L/2 + 128·L/2).

### 3.4 not ReplaySSM

ReplaySSM (`cache_config.use_replayssm`, `replayssm_buffer_len=16`; PRs [#48018](https://github.com/aarnphm/aarnphm.github.io/issues/48018) by Johnny Liou, [#52506](https://github.com/aarnphm/aarnphm.github.io/issues/52506) and [#52928](https://github.com/aarnphm/aarnphm.github.io/issues/52928) by Andrii Skliar) is a Mamba2 decode kernel. It caches the last B SSM _inputs_ and skips per-step full-state stores, writing the checkpoint on flush (`config/cache.py:203-214`). It reconstructs exact recurrent state for spec-decode rollback and prefix-cache alignment. SWA bounded replay instead recomputes _tokens_ to rebuild a truncated attention window, is approximate, and is DSv4.1-only. The two share only the word “replay”.

### 3.5 P/D and offload

- `KVConnectorBase.get_loaded_kv_cache_group_ids` defaults to the prefix-cacheable groups, i.e. what a hash-addressed store holds (`kv_connector/v1/base.py:490-498`). NIXL returns every transfer group, SWA included (`nixl/connector.py:170-171`).
- The scheduler’s `_load_restores_replay_window` checks that set ⊇ the replay groups (`sched/scheduler.py:3104-3111`). If it holds: no replay, no block-trim, `replay_start=0` ([#59197](https://github.com/aarnphm/aarnphm.github.io/issues/59197)).
- Consequence: NIXL/Mooncake P/D loads take the window as is. Offload/store connectors (SimpleCPU, Mooncake store, tiering) hold no window, so D replays the last 128.
- Prefiller backoff: with bounded replay, NIXL/Mooncake make P compute only N−1 prompt tokens, as for Mamba. The sliding window “is not replayed after a P/D load, so it must be laid out for the token the decoder recomputes” (`nixl/base_scheduler.py:379-395`, `mooncake_connector.py:946-990`).
- SWA transfer groups are clipped to the last `cdiv(128/bs)+1` blocks (`mooncake_connector.py:916-944`).
- Engram’s slot `hash_cache` is not part of the KV cache, so a P/D load leaves it unwritten. MRv2 supplies `lookback_token_ids` from device token history, which take precedence (`engram.py:27-35`).
- Heterogeneous TP: the compressed / index / SWA caches are all `num_kv_heads=1` (`max_tp_shards=1` on SWA), so they are replicated across TP ranks. Heterogeneous-TP P/D therefore moves whole-latent pages, with no head split (inferred from the specs; I did not trace NIXL’s descriptor code for V4.1).

---

See [[thoughts/inference/deepseek-v4.1-serving|serving notes]] for model execution and PR history.

