---
title: '42082'
pageLayout: default
slug: thoughts/inference/vllm-prs/42082
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/42082.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#42082](https://github.com/aarnphm/aarnphm.github.io/issues/42082) \[RFC\]: Standardize KV-cache Layouts

[vllm-project/vllm#42082](https://github.com/vllm-project/vllm/issues/42082) · LucasWilkinson · OPEN · created 2026-05-08T15:05:38Z

# Motivation

Right now attention backends can have semantically different and physically different KV-cache layouts. This leads to messy KV-connector code (i.e. lots of `is_mamba` and `is_mla` flags), and creates a tight coupling between KV-connector and the attention backends, e.g. <https://github.com/vllm-project/vllm/blob/4140faa4a51d42cb9618949bee28fd47682f611c/vllm/v1/attention/backends/flash_attn.py#L152-L171>

This also leads to other related bugs like: [vllm-project/vllm#41657 (comment)](https://github.com/vllm-project/vllm/pull/41657#issuecomment-4400641394)

More details in: <https://docs.google.com/document/d/1-TnWZf8jI6nWgE-xaQkgsWy130CVUqqt1NYqY-mAOe4/edit?usp=sharing>

# Proposed Change

## Standard Logical Shape



```
[num_layers_slots, num_blocks, num_heads, num_states, <state_content>]
```

Where:

- **num_states** — token positions per block (or 1 for recurrent state)
- **num_heads** — heads (or 1 for headless backends like MLA)
- **\<state_content>** — backend-specific, always contiguous

Every backend maps to this shape:

| Backend                                 | num_states     | num_heads      | state_content             |
| :-------------------------------------- | :------------- | :------------- | :------------------------ |
| GQA                                     | `block_size`   | `num_kv_heads` | `[2, head_dim]`           |
| DeepSeek V4 (c4a, 1 layer type example) | `block_size/4` | 1              | `[latent_size]`           |
| MLA                                     | `block_size`   | 1              | `[latent_size]`           |
| Mamba2 Conv                             | 1              | 1              | `[conv_dim/tp, kernel-1]` |
| Mamba2 SSM                              | 1              | `num_heads`    | `[head_dim, state_size]`  |

---

### Stride Order: Logical Shape vs Physical Layout

The **semantic shape** is always `[layer_slots, blocks, heads, states, <content>]`. The **physical layout** (memory order) is controlled by a `stride_order` permutation.

**LBHNC** (**default**, hetero-TP-friendly):



```
Logical:   [L, B, H, N, C]
Physical:  [L, B, H, N, C]  (identity)
stride_order = (0, 1, 2, 3, 4)
```

Head slices are contiguous in memory. Good for head-level TP transfers, hetero-TP disagg.

**LBNHC** (hetero-block-size-friendly):



```
Logical:   [L, B, H, N, C]
Physical:  [L, B, N, H, C]
stride_order = (0, 1, 3, 2, 4)
```

State slices are contiguous in memory. Good for block-level transfers, hetero-block-size disagg.

**BLHNC** (larger P/D transfers when homo-TP/block-size):



```
Logical:   [L, B, H, N, C]
Physical:  [B, L, H, N, C]
stride_order = (1, 0, 2, 3, 4)
```

All layers’ data for one block is contiguous. The connector can transfer an entire block across all layers in a single RDMA read.

**BHLNC** (larger P/D transfers for hetero-TP):



```
Logical:   [L, B, H, N, C]
Physical:  [B, H, L, N, C]
stride_order = (1, 2, 0, 3, 4)
```

For a given block and head, all layers are contiguous. One contiguous region per (block, head) across all layers.

> **Layout constraints for shared tensors:** When using HMA and groups share the same underlying `KVCacheTensor` (because they have the same `page_size_bytes`), the layers in that tensor may have different H, N, or C dimensions. In that case:
>
> 1. The per-layer-per-block content, \[H, N, C\], must be contiguous in physical memory (i.e. `assert layout[0, 0, ...].is_contiguous()  # [0, 0, H, N, C]`). This rules out `LBNHC` and `BHLNC`; only `LBHNC` and `BLHNC` satisfy this constraint.
> 2. If the H, N, and C sizes match across all layers in the tensor, any layout (including `LBNHC` and `BHLNC`) is valid.
>
> For example, DeepSeek V4 has mixed compression ratios producing layers with different N and C sizes sharing a tensor — the default `LBHNC` layout handles this correctly.

---

### KVCacheSpec Properties



```py
@dataclass
class KVCacheSpec:
  num_heads: int  # heads (1 if headless)
  tokens_per_state: int  # -1 infinite, 1 standard, N compressed
  state_content_size: int  # bytes per state per head
```

NOTE: `sliding_window` and `attention_chunk_size` will still be needed in the spec but we plan to refactor those away in a subsequent attention backends refactor .

`num_states` is derived from `block_size / tokens_per_state` at allocation time — it’s not known when the spec is constructed.

| Backend     | num_heads      | tokens/state | num_states (derived) |
| :---------- | :------------- | :----------- | :------------------- |
| GQA         | `num_kv_heads` | 1            | `block_size`         |
| DeepSeek V4 | `num_kv_heads` | 4            | `block_size/4`       |
| MLA         | 1              | 1            | `block_size`         |
| Mamba2 Conv | 1              | -1           | 1                    |
| Mamba2 SSM  | `num_heads`    | -1           | 1                    |

---

### Spec-Driven TP Slicing

Instead of connector byte math, **specs provide their own slicing**:



```py
class KVCacheSpec:
  def slice_for_tp_transfer(
    self, my_tensor, my_tp, my_rank, other_spec, other_tp, other_rank
  ) -> list[Tensor]:
    """Return slices of `my_tensor` that need to be transferred.
    Takes both sides' TP configs since tp_size is a deployment
    property, not part of the spec."""
    ...
```

Each spec subclass knows how to slice its own tensor. The signature takes the other spec so it can compute the correct overlap — GQA slices by heads, Mamba slices by x/B/C components, MLA may be a single unsliced latent.

#### GQA Slicing

Head slicing handling replication on either side (or both)



```py
def slice_for_tp_transfer(
  self, src_tensor, src_tp, src_rank, dst_spec, dst_tp, dst_rank
):

  src_heads = src_tp * self.num_heads
  dst_heads = dst_tp * dst_spec.num_heads

  if dst_heads > src_heads:
    dst_rank %= src_tp
  elif src_rank >= dst_tp:
    return []

  head_range = lambda tp_rank, num_heads: (
    tp_rank * num_heads,
    (tp_rank + 1) * num_heads,
  )

  src_head_range = head_range(src_rank, self.num_heads)
  dst_head_range = head_range(dst_rank, dst_spec.num_heads)

  overlap = get_overlap(src_head_range, dst_head_range)
  if overlap is None:
    return []

  # Convert global overlap to local tensor indices
  local_slice = slice(
    overlap.start - src_head_range[0], overlap.stop - src_head_range[0]
  )
  return [src_tensor[:, :, local_slice]]


def get_overlap(range1, range2):
  # Calculate the intersection boundaries
  overlap_start = max(range1[0], range2[0])
  overlap_end = min(range1[1], range2[1])

  # Check if the boundaries form a valid range
  if overlap_start <= overlap_end:
    return slice(overlap_start, overlap_end)
  return None
```

#### Mamba Conv Slicing: 3 Slices for x/B/C



```py
def slice_for_tp_transfer(
  self, my_tensor, my_tp, my_rank, other_spec, other_tp, other_rank
):
  # tensor: [layers, blocks, 1, 1, conv_dim/tp, kernel-1]
  x_range = self._compute_x_range(my_tp, my_rank, other_tp, other_rank)
  b_range = self._compute_b_range(my_tp, my_rank, other_tp, other_rank)
  c_range = self._compute_c_range(my_tp, my_rank, other_tp, other_rank)
  return [
    my_tensor[:, :, 0, 0, x_range, :],  # x slice
    my_tensor[:, :, 0, 0, b_range, :],  # B slice
    my_tensor[:, :, 0, 0, c_range, :],  # C slice
  ]
```

Same interface — spec knows its own structure.

---

## Connector Becomes Generic

The connector needs both source and destination specs — if they have different stride orders (e.g., prefill uses HND, decode uses NHD), the source and destination slices map to different physical memory regions. Both specs produce slices in the same semantic order, so they pair directly. No backend-specific code in the connector.



```py
# TP slicing (once at setup — determines which heads to transfer)
src_slices = src_spec.slice_for_tp_transfer(
  src_cache, src_tp, src_rank, dst_spec, dst_tp, dst_rank
)
dst_meta = make_meta_kv_cache(dst_spec, num_blocks, dst_block_size)
dst_slices = dst_spec.slice_for_tp_transfer(
  dst_meta, dst_tp, dst_rank, src_spec, src_tp, src_rank
)

# Per-block transfer (handles hetero block sizes naturally)
# blk_idx is an int, states_slice is a slice
for (
  src_blk_idx,
  src_states_slice,
  dst_blk_idx,
  dst_states_slice,
) in block_mapping:
  for src_s, dst_s in zip(src_slices, dst_slices):
    src_block = src_s[
      :, src_blk_idx, :, src_states_slice, ...
    ]  # still [num_layers, heads_slice, state_slice, <state_content>]
    dst_block = dst_s[
      :, dst_blk_idx, :, dst_states_slice, ...
    ]  # still [num_layers, heads_slice, state_slice, <state_content>]
    for src_chunk, dst_chunk in jointly_contiguous_chunks(
      src_block, dst_block
    ):
      self._rdma_transfer(src_chunk, dst_chunk)
```

TP slicing is computed once (which heads overlap). The \``` block_mapping` `` hetero block sizes. `jointly_contiguous_chunks` decomposes each pair into regions contiguous in both physical layouts; and will naturally take advantage (if possible) of layers that are packed in an interleaved fashion . Destination offsets come from a meta tensor (`device='meta'`, no GPU memory).

Connector is backend-agnostic.

### Feedback Period.

_No response_

### CC List.

@heheda12345 @benchislett @NickLucche @ZhanqiuHu

### Worktracking

prepatory:

- [vllm-project/vllm#42122](https://github.com/vllm-project/vllm/pull/42122)
- [vllm-project/vllm#42095](https://github.com/vllm-project/vllm/pull/42095)

next steps

- \[1/N\] [vllm-project/vllm#42374](https://github.com/vllm-project/vllm/pull/42374) (standardize attention backends and kv-cache allocation, remove `get_kv_cache_stride_order` and `get_kv_cache_shape`)
- \[2/N\] [vllm-project/vllm#43151](https://github.com/vllm-project/vllm/pull/43151)
- \[3/N\] refactor the kv-connector code to use meta-tensors and slicing

---

## comments

**orozery** (2026-05-11T11:53:17Z):

Most KVConnector copy per-block (across all layers) or per-head.
So having the `num_layers` dimension AFTER `num_blocks` or even after `num_heads` is more desirable for performance reasons.

I do think that the worker-side connector should not care about the attention backend.
Right now it uses `get_kv_cache_shape` to understand the shape of the given tensors.
My suggestion is to change the `register_kv_caches` method to get a canonical representation of KV cache tensors,
where each tensor is shaped `(num_blocks, page_size_in_bytes)` with int8 dtype.
e.g. for flash-attention layer, each layer would be split to 2 tensors (one for k and one for v).
We can augment this with `num_heads_stride` to allow connectors to locate specific head data. (similarly `num_blocks_stride`).

We actually implemented it in this open PR [#37885](https://github.com/aarnphm/aarnphm.github.io/issues/37885).

**LucasWilkinson** (2026-05-14T12:32:36Z):

> Most KVConnector copy per-block (across all layers) or per-head.
> So having the num_layers dimension AFTER num_blocks or even after num_heads is more desirable for performance reasons.

I believe this is `BLNHC` and `BHLNC` layouts above; is there something im missing?

**zyongye** (2026-05-14T14:23:30Z):

cc @ivanium. I think DSV4 layout is way more complicated than what is written given different compressor ratio

**orozery** (2026-05-14T15:28:16Z):

> I believe this is `BLNHC` and `BHLNC` layouts above; is there something im missing?

I missed the `Logical` part in `Standard Logical Shape` :)

**LucasWilkinson** (2026-05-15T03:21:45Z):

> cc [@ivanium](https://github.com/ivanium). I think DSV4 layout is way more complicated than what is written given different compressor ratio

Apologies I should have clarified that was an example for just one layer, that was my bad; updated the RFC. Good call on being more explicit on DSv4, I added this section

> **Layout constraints for shared tensors:** When using HMA and groups share the same underlying `KVCacheTensor` (because they have the same `page_size_bytes`), the layers in that tensor may have different H, N, or C dimensions. In that case:
>
> 1. The per-layer-per-block content, \[H, N, C\], must be contiguous in physical memory (i.e. `assert layout[0, 0, ...].is_contiguous()  # [0, 0, H, N, C]`). This rules out `NHC` and `BHLNC`; only `HNC` and `BLHNC` satisfy this constraint.
> 2. If the H, N, and C sizes match across all layers in the tensor, any layout (including `NHC` and `BHLNC`) is valid.
>
> For example, DeepSeek V4 has mixed compression ratios producing layers with different N and C sizes sharing a tensor — the default `HNC` layout handles this correctly.

to the RFC to help clarify; but please do let me know if I am missing something!

**ApostaC** (2026-06-04T20:07:23Z):

@LucasWilkinson Can we still keep some standardized information (enum or string) to indicate how to parse the `state_content`?
It will be critical for researchers to parse the format and apply optimizations, such as quantization (which will need to extract K and V separately).

Ideally, K and V should be separated for researchers.

**KuntaiDu** (2026-06-04T21:08:16Z):

> [@LucasWilkinson](https://github.com/LucasWilkinson) Can we still keep some standardized information (enum or string) to indicate how to parse the `state_content`? It will be critical for researchers to parse the format and apply optimizations, such as quantization (which will need to extract K and V separately).
>
> Ideally, K and V should be separated for researchers.

Agreed. Recent papers start to quantize K and V differently (example: <https://dl.acm.org/doi/10.1145/3731569.3764810> ). It would be ideal if we still keep K and V.

As for TP slicing --- I am thinking of a slightly different approach. Basically:
KV cache producer side ALWAYs slicing KV caches by TP8 / TP16
KV consumer side re-assembles from TP8 / TP16 to target TP (e.g. TP4).
By doing this, KV producer always slice the KV caches, and KV consumer always merge KV caches, and potentially make the code easier to maintain.

**NickLucche** (2026-06-05T10:02:45Z):

> KV producer always slice the KV caches, and KV consumer always merge KV caches, and potentially make the code easier to maintain.

I don’t think I get this sorry, the `slice_for_tp_transfer` is basically a porting of the logic contained in the NixlConnector to deal with heteroTP configurations @KuntaiDu

**LucasWilkinson** (2026-06-05T14:20:32Z):

> [@LucasWilkinson](https://github.com/LucasWilkinson) Can we still keep some standardized information (enum or string) to indicate how to parse the `state_content`? It will be critical for researchers to parse the format and apply optimizations, such as quantization (which will need to extract K and V separately).
>
> Ideally, K and V should be separated for researchers.

Separate KV complicates the KV-connectors quite a bit; we are trying to abstract the KV-connectors so they now longer need to aware for Mamba/Linear vs MLA vs GQA vs what-ever-weird-future-attention. Generally for KV-quantization we recommend you right a custom backend (like TurboQuant) and can re-interpret content however you want. After this and [vllm-project/vllm#42449](https://github.com/vllm-project/vllm/issues/42449) writing custom backends should be dramatically simplified. Does this solve what you talking about or are you trying to capture information outside the backends?

> As for TP slicing --- I am thinking of a slightly different approach. Basically:
> KV cache producer side ALWAYs slicing KV caches by TP8 / TP16
> KV consumer side re-assembles from TP8 / TP16 to target TP (e.g. TP4).
> By doing this, KV producer always slice the KV caches, and KV consumer always merge KV caches, and potentially make the code easier to maintain.

Like @NickLucche said im not sure this addresses we are trying to solve here which is trying to make “how to slice for TP” pluggable by the model code/definition. This is intended to make OOT model support and day-0 support much easier by preventing use from having to touch the KV-connectors to support a new attention type (this has been a major pain point with the rise of hybrid models).

**maobaolong** (2026-10-26T07:40:10Z):

One concrete OOT case: vLLM-RBLN registers `[NB, 2, H, 1, N, D]` or `[2, NB, H, 1, N, D]`, depending on the kernel. At `NB == 2`, both shape and contiguous strides are identical, so LMCache can’t recover which axis means blocks.

I’ve opened [#63699](https://github.com/aarnphm/aarnphm.github.io/issues/63699) to pass optional, producer-declared axes with the actual registered views. The standard allocator declares `[B, H, N, C]`; custom allocators can declare their own axes. Existing connectors keep their tensor-only hook, and MultiConnector forwards the metadata.

This complements the allocator work here: `KVCacheLayout` still describes physical stride order, and `CONTENT` stays opaque with encoding owned by the cache spec. The [RBLN PoC](https://github.com/RBLN-SW/vllm-rbln/pull/1297) shows the producer side; [LMCache#5866](https://github.com/LMCache/LMCache/issues/5866) tracks the consumer work.

