---
title: '44455'
pageLayout: default
slug: thoughts/inference/vllm-prs/44455
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/44455.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#44455](https://github.com/aarnphm/aarnphm.github.io/issues/44455) \[2/N\]\[KV-Cache Layout Refactor\] Pack K/V into the content dim across attention backends

[vllm-project/vllm#44455](https://github.com/vllm-project/vllm/pull/44455) · LucasWilkinson · MERGED · created 2026-06-03T22:22:32Z · merged 2026-07-11T15:11:16Z

PR [#42374](https://github.com/aarnphm/aarnphm.github.io/issues/42374) (first part of RFC [#42082](https://github.com/aarnphm/aarnphm.github.io/issues/42082)) has been split into 4 PRs:

[#44454](https://github.com/aarnphm/aarnphm.github.io/issues/44454) \[1/N\]\[KV-Cache Layout Refactor\] Refactor DSV4 KV cache config
<span>&rarr;</span> [#44455](https://github.com/aarnphm/aarnphm.github.io/issues/44455) \[2/N\]\[KV-Cache Layout Refactor\] Pack K/V into the content dim across attention backends
[#44456](https://github.com/aarnphm/aarnphm.github.io/issues/44456) \[3/N\]\[KV-Cache Layout Refactor\] Standardize Mamba cache; drop `get_transfer_cache_regions`
[#44458](https://github.com/aarnphm/aarnphm.github.io/issues/44458) \[4/N\]\[KV-Cache Layout Refactor\] Standardize KV cache layout

## Summary

Packs K and V into the content dimension across attention backends, changing the KV cache shape from `(num_blocks, 2, block_size, num_kv_heads, head_size)` (separate leading K/V dim) to `(num_blocks, num_kv_heads, block_size, 2*head_size)`. Backends recover K/V via `kv_cache.transpose(1, 2).split(head_size, dim=-1)`. This removes the need for `get_kv_cache_stride_order()` in most paths and is a major diff reduction toward [#42374](https://github.com/aarnphm/aarnphm.github.io/issues/42374).

Stacked on top of [#44454](https://github.com/aarnphm/aarnphm.github.io/issues/44454) (land that first)

## Testing



```bash
pytest tests/v1/attention/test_attention_backends.py -v
pytest tests/kernels/attention/test_cache.py -v
pytest tests/v1/worker/test_gpu_model_runner.py -v
```

Most backend paths require specific GPUs (ROCm, nvfp4/flashinfer); the submitter will run the relevant subset on appropriate hardware before merge.

## AI assistance

This PR was prepared with AI assistance (Claude).

---

## comments

**mergify** (2026-06-04T22:00:16Z):

Documentation preview: [https://vllm—44455.org.readthedocs.build/en/44455/](https://vllm--44455.org.readthedocs.build/en/44455/)

**mergify** (2026-06-05T11:53:33Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @LucasWilkinson.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**mergify** (2026-06-08T03:31:34Z):

Hi @LucasWilkinson, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

**mergify** (2026-06-12T11:21:58Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @LucasWilkinson.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**AndreasKaratzas** (2026-06-15T17:22:09Z):

The v1 sample on amd is a known failure and is expected to be resolved EOD. But the other one is not.

**mergify** (2026-06-16T12:10:09Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @LucasWilkinson.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**Dao007forever** (2026-06-16T17:11:36Z):

Hi @LucasWilkinson , I was running this branch (with merged from `main`) vs `main` in PD setup for Minimax M3. There is a bug somewhere which corrupted the transfer:



```
┌────────────────────┬───────────────────┬───────────────────────────────────────┐
  │       Branch       │    Correctness    │                Output                 │
  ├────────────────────┼───────────────────┼───────────────────────────────────────┤
  │ testm3 (1a50784ed) │ 0/8 = 0.00 — FAIL │ total gibberish (corrupted KV)        │
  ├────────────────────┼───────────────────┼───────────────────────────────────────┤
  │ main (0a1c5034f)   │ 6/8 = 0.75 — PASS │ coherent + correct (48+24=72 #### 72) │
  └────────────────────┴───────────────────┴───────────────────────────────────────┘
```

Commands to run:



```
Prefill:
vllm serve MiniMaxAI/MiniMax-M3
          --trust-remote-code
          --language-model-only
          --load-format auto
          -tp 4
          -O3
          --block-size 128
          --attention-backend FLASH_ATTN
          --max-model-len 1184
          --enforce-eager
          --gpu-memory-utilization 0.9
          --enable-prefix-caching
          --enable-chunked-prefill
          --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer","kv_load_failure_policy":"fail","kv_buffer_device":"cuda","kv_connector_extra_config":{"enforce_handshake_compat":false}}'

Decode:
vllm serve MiniMaxAI/MiniMax-M3
          --trust-remote-code
          --language-model-only
          --load-format auto
          -tp 1
          -dp 4
          -ep
          --data-parallel-size-local 4
          --data-parallel-hybrid-lb
          -O3
          --block-size 128
          --attention-backend FLASH_ATTN
          --max-model-len 1184
          --enforce-eager
          --gpu-memory-utilization 0.9
          --enable-prefix-caching
          --enable-chunked-prefill
          --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer","kv_load_failure_policy":"fail","kv_buffer_device":"cuda","kv_connector_extra_config":{"enforce_handshake_compat":false}}'
```

**LucasWilkinson** (2026-06-16T17:13:34Z):

> Hi @LucasWilkinson , I was running this branch (with merged from `main`) vs `main` in PD setup for Minimax M3. There is a bug somewhere which corrupted the transfer:
>
>
>
> ```
>
> ┌────────────────────┬───────────────────┬───────────────────────────────────────┐
>
>   │       Branch       │    Correctness    │                Output                 │
>
>   ├────────────────────┼───────────────────┼───────────────────────────────────────┤
>
>   │ testm3 (1a50784ed) │ 0/8 = 0.00 — FAIL │ total gibberish (corrupted KV)        │
>
>   ├────────────────────┼───────────────────┼───────────────────────────────────────┤
>
>   │ main (0a1c5034f)   │ 6/8 = 0.75 — PASS │ coherent + correct (48+24=72 #### 72) │
>
>   └────────────────────┴───────────────────┴───────────────────────────────────────┘
>
> ```

@Dao007forever Thanks for testing it! Can you please provide the reproduction commands?

**LucasWilkinson** (2026-06-17T03:52:40Z):

@Dao007forever should be fixed now, thanks for the heads up! (It was mostly an artifact of MiniMax-M3 landing after this refactor was started)

**ZhanqiuHu** (2026-06-18T15:43:56Z):

**E2E lm-eval PD disagg tested** (block_size=128, NIXL connector):

**Passed:**

- `Qwen/Qwen3-0.6B` × \[1p1d, 2p2d, 4p4d, 1p2d, 2p1d, 1p4d, 4p1d, 2p4d, 4p2d\] × \[FLASH_ATTN, FLASHINFER, TRITON_ATTN\]
- `deepseek-ai/DeepSeek-V2-Lite` × \[1p1d, 2p2d, 4p4d, 1p2d, 2p1d, 1p4d, 4p1d, 2p4d, 4p2d\] × FLASH_ATTN
- `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8` × \[1p1d, 2p2d, 4p4d, 1p2d, 2p1d, 1p4d, 4p1d, 2p4d, 4p2d\] × \[FLASHINFER, FLASH_ATTN\]
- `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8` × \[1p1d, 2p2d, 4p4d, 1p2d, 2p1d\] × TRITON_ATTN

**Unsupported (pre-existing):**

- `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8` × TRITON_ATTN × \[1p4d, 4p1d, 2p4d, 4p2d\] block-size mismatch between P and D

**mergify** (2026-06-23T18:05:11Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @LucasWilkinson.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**mergify** (2026-06-24T11:06:44Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @LucasWilkinson.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**mergify** (2026-06-25T15:05:29Z):

Hi @LucasWilkinson, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

**mergify** (2026-06-26T07:50:12Z):

Documentation preview: [https://vllm—44455.org.readthedocs.build/en/44455/](https://vllm--44455.org.readthedocs.build/en/44455/)

**mergify** (2026-06-26T09:20:16Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @LucasWilkinson.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**mergify** (2026-06-30T07:00:49Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @LucasWilkinson.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**AndreasKaratzas** (2026-07-01T22:54:54Z):

@Rohan138 @tjtanaa PTAL

**mergify** (2026-07-02T20:19:28Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @LucasWilkinson.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**mergify** (2026-07-03T20:13:51Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @LucasWilkinson.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**mergify** (2026-07-04T04:21:48Z):

Hi @LucasWilkinson, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

**mergify** (2026-07-09T00:23:19Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @LucasWilkinson.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

