---
title: '27743'
pageLayout: default
slug: thoughts/inference/vllm-prs/27743
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/27743.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#27743](https://github.com/aarnphm/aarnphm.github.io/issues/27743) \[KVConnector\]\[Core\] Support cross-layer KV blocks

[vllm-project/vllm#27743](https://github.com/vllm-project/vllm/pull/27743) · orozery · MERGED · created 2025-10-29T12:27:23Z · merged 2025-11-20T18:09:59Z

Core of RFC [#27742](https://github.com/aarnphm/aarnphm.github.io/issues/27742).
Following this PR, connectors can turn-on and adapt to the new layout.

This PR enables the GPU model runner to allocate the KV cache tensors, so that the KV data for all layers will be contiguous per block. This can yield a significant speed up the transfer time of KV transfers (e.g. X4), such in the case of using NixlConnector or OffloadingConnector. Currently, this new layout is disabled by default, and will only be enabled when using a connector which explicitly prefers this new layout. Also, this new layout is currently only supported for uniform (non HMA) models.

---

## comments

**mergify** (2025-10-29T12:28:05Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @orozery.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**ApostaC** (2025-10-29T17:43:09Z):

Quick question:

- How should the connector know whether the layout is the new one or the old one?
- Should the connector have 2 different code paths to support both layouts? This will potentially increase the connector code by a lot.

**orozery** (2025-10-29T17:53:14Z):

> - How should the connector know whether the layout is the new one or the old one?

My initial thought is to check on `register_kv_caches` whether two layer tensors have the same `.storage().data_ptr()`.
Of course we can also expose a new API.

> - Should the connector have 2 different code paths to support both layouts? This will potentially increase the connector code by a lot.

Connectors don’t need to support the new layout if they don’t want to.
For the offloading connector and nixl it does not seem like a big change.
For the offloading connector for example this is just writing a new cuda transfer function.

**orozery** (2025-10-30T06:44:46Z):

> Like the high-level idea, but I think we need to further discuss the design.
>
> Also, the correctness of this PR is not tested. Will the new code have any correctness & performance impact on the underlying attention modules?

Good point.
I was not sure myself this will work, so I just ran some “how much is two plus two” prompt to verify this works (for flash attention and flash infer).
Is there a way to make a CI test out of it? Or some other way?

**ApostaC** (2025-10-30T20:29:46Z):

> Is there a way to make a CI test out of it? Or some other way?

I think lm_eval can be used for correctness benchmarks.

**mergify** (2025-11-02T09:27:56Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @orozery.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**orozery** (2025-11-06T14:00:18Z):

@NickLucche I’ve appended the changes of the offloading connector utilizing the new layout.
For 2 reasons (as you suggested):

1. Verifying the correctness of the new layout (which is only enabled per-connector request).
2. Testing performance with the new layout.

For (1) I’ve added a basic test (`_accuracy_test` inside test_cpu_offloading.py).
For (2), there are 2 cases:

1. Performance impact on KV transfer time. This is already measured today in test_cpu_offlading.py.
   The results show drastic improvement in latency. For example (output taken from running test_cpu_offloading.py):
   Current layout:



```
Average times:
    Cold: 147.80ms
    GPU hit: 9.23ms
    CPU hit: 109.41ms
```

New layout:



```
Average times:
    Cold: 145.48ms
    GPU hit: 9.03ms
    CPU hit: 43.14ms
```

2. Performance impact on model execution. I did notice any significant change in performance. I ran (once with flashinfer, and once with flash attention):



```
vllm serve meta-llama/Llama-3.1-8B-Instruct --no-enable-prefix-caching --kv-transfer-config '{"kv_connector": "OffloadingConnector", "kv_role": "kv_both", "kv_connector_extra_config": {"block_size": 1048576, "num_cpu_blocks": 1}}'
```

Followed by:



```
vllm bench serve --dataset-name sharegpt --model meta-llama/Llama-3.1-8B-Instruct --dataset-path ./ShareGPT_V3_unfiltered_cleaned_split.json
```

For flash-attention:
Current layout:



```
============ Serving Benchmark Result ============
Successful requests:                     1000
Failed requests:                         0
Benchmark duration (s):                  56.65
Total input tokens:                      215196
Total generated tokens:                  197247
Request throughput (req/s):              17.65
Output token throughput (tok/s):         3481.67
Peak output token throughput (tok/s):    6495.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          7280.17
---------------Time to First Token----------------
Mean TTFT (ms):                          17969.26
Median TTFT (ms):                        16236.38
P99 TTFT (ms):                           42501.62
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          67.87
Median TPOT (ms):                        64.01
P99 TPOT (ms):                           147.37
---------------Inter-token Latency----------------
Mean ITL (ms):                           58.03
Median ITL (ms):                         52.68
P99 ITL (ms):                            154.11
==================================================
```

New layout:



```
============ Serving Benchmark Result ============
Successful requests:                     1000
Failed requests:                         0
Benchmark duration (s):                  57.63
Total input tokens:                      215196
Total generated tokens:                  198186
Request throughput (req/s):              17.35
Output token throughput (tok/s):         3438.83
Peak output token throughput (tok/s):    6758.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          7172.80
---------------Time to First Token----------------
Mean TTFT (ms):                          18064.90
Median TTFT (ms):                        16198.84
P99 TTFT (ms):                           42469.16
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          68.03
Median TPOT (ms):                        63.70
P99 TPOT (ms):                           148.00
---------------Inter-token Latency----------------
Mean ITL (ms):                           57.78
Median ITL (ms):                         50.87
P99 ITL (ms):                            157.69
==================================================
```

For flash-infer:
Current layout:



```
============ Serving Benchmark Result ============
Successful requests:                     1000
Failed requests:                         0
Benchmark duration (s):                  49.95
Total input tokens:                      215196
Total generated tokens:                  197970
Request throughput (req/s):              20.02
Output token throughput (tok/s):         3963.21
Peak output token throughput (tok/s):    6720.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          8271.26
---------------Time to First Token----------------
Mean TTFT (ms):                          15721.31
Median TTFT (ms):                        14397.39
P99 TTFT (ms):                           35988.98
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          58.00
Median TPOT (ms):                        52.94
P99 TPOT (ms):                           138.76
---------------Inter-token Latency----------------
Mean ITL (ms):                           49.15
Median ITL (ms):                         40.10
P99 ITL (ms):                            140.68
==================================================
```

New layout:



```
============ Serving Benchmark Result ============
Successful requests:                     1000
Failed requests:                         0
Benchmark duration (s):                  49.83
Total input tokens:                      215196
Total generated tokens:                  197685
Request throughput (req/s):              20.07
Output token throughput (tok/s):         3966.81
Peak output token throughput (tok/s):    6816.00
Peak concurrent requests:                1000.00
Total Token throughput (tok/s):          8285.01
---------------Time to First Token----------------
Mean TTFT (ms):                          15633.70
Median TTFT (ms):                        14393.79
P99 TTFT (ms):                           35934.41
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          57.75
Median TPOT (ms):                        53.03
P99 TPOT (ms):                           138.87
---------------Inter-token Latency----------------
Mean ITL (ms):                           49.10
Median ITL (ms):                         40.00
P99 ITL (ms):                            141.52
==================================================
```

**mergify** (2025-11-19T12:36:58Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @orozery.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**NickLucche** (2025-11-19T14:13:12Z):

@ApostaC I think you need to unblock too

**ApostaC** (2025-11-20T06:10:10Z):

@NickLucche @heheda12345 I’m okay to proceed if you guys want to.
However, I do have this strong concern about code understandability [vllm-project/vllm#27743 (comment)](https://github.com/vllm-project/vllm/pull/27743#discussion_r2544443777) . I feel like it will be very easy to make mistakes if I’m a contributor who only focus on attn backends without knowing this KV cache layout thing.

**NickLucche** (2025-11-20T09:45:37Z):

I see your point @ApostaC , imo I think the confusion mainly stems from the `include_num_layers_dimension` name: the intent is to return a transfer-optimized layout, that includes the layer dim but not finalized to it.
But I would like to enforce @orozery point on this actually being an optional feature for an attn backend, as well as the previous `get_kv_cache_stride_order ` interface (which I introduced in v1 iirc) already relying on other backends implementation in code as reference, rather than providing clear docs guidance.

**TheTinyTeddy** (2026-01-22T02:16:47Z):

@orozery Could I ask what performance impact will the contiguous kv block (layer-wise and K\&V-wise) have on the main GPU computation (e.g. attention) in the case of not using KV offload?

I was wondering why not use this as default in the first place, was it purely due to coding convenience?

**orozery** (2026-01-22T07:07:07Z):

> @orozery Could I ask what performance impact will the contiguous kv block (layer-wise and K\&V-wise) have on the main GPU computation (e.g. attention) in the case of not using KV offload?

From my experiments, it does not effect compute.
See here:
[vllm-project/vllm#27743 (comment)](https://github.com/vllm-project/vllm/pull/27743#issuecomment-3497400950)

> I was wondering why not use this as default in the first place, was it purely due to coding convenience?

I also asked myself the same question :)
I guess it was coding convenience.
This per-layer allocation was actually introduced in one of the first commits of vLLM:
[vllm-project/vllm@`e7bee2a`](https://github.com/vllm-project/vllm/commit/e7bee2aa811963b8c5ce352a427595749f6bfca1)
So I guess @WoosukKwon can answer that.

