---
title: '33339'
pageLayout: default
slug: thoughts/inference/vllm-prs/33339
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/33339.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#33339](https://github.com/aarnphm/aarnphm.github.io/issues/33339) Enable Cross layers KV cache layout at NIXL Connector V2

[vllm-project/vllm#33339](https://github.com/vllm-project/vllm/pull/33339) · liranschour · MERGED · created 2026-01-29T12:06:30Z · merged 2026-02-05T10:17:03Z

## Purpose

Enable NIXL Connector to us the new continuous cross layer KV cache layout described in [RFC](https://github.com/vllm-project/vllm/issues/27742) and implemented in [#27743](https://github.com/vllm-project/vllm/pull/27743)

Demonstrate performance improvement of more the 2x in Tok/sec and TTFT due to dramatic reduction of fragmentation of transfer buffers.

Tested with P!=D with run_accuracy_test.sh P=1 D=2

| branch          | num reqs | input len | TTFT     | ITL   | tok/s     | Desc/transfer |
| --------------- | -------- | --------- | -------- | ----- | --------- | ------------- |
| main            | 1000     | 16        | 18756.42 | 5.35  | 5288.41   | 56            |
| kv_cross_layers | 1000     | 16        | 8494.44  | 8.74  | 8572.56   | 1             |
| main            | 128      | 1024      | 1660.84  | 9.40  | 37945.20  | 3528          |
| kv_cross_layers | 128      | 1024      | 686.98   | 9.26  | 55418.76  | 1             |
| main            | 128      | 10240     | 11140.52 | 42.78 | 62339.74  | 34000         |
| kv_cross_layers | 128      | 10240     | 5226.71  | 14.41 | 117631.48 | 422           |

## Test Plan

## Test Result

---

<details>
<summary> Essential Elements of an Effective PR Description Checklist </summary>

- [ ] The purpose of the PR, such as “Fix some issue (link existing issues this PR will resolve)”.
- [ ] The test plan, such as providing test command.
- [ ] The test results, such as pasting the results comparison before and after, or e2e results
- [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
- [ ] (Optional) Release notes update. If your change is user facing, please update the release notes draft in the [Google Doc](https://docs.google.com/document/d/1YyVqrgX4gHTtrstbq8oWUImOyPCKSGnJ7xtTpmXzlRs/edit?tab=t.0).

</details>

---

## comments

**mergify** (2026-01-29T12:07:09Z):

Documentation preview: [https://vllm—33339.org.readthedocs.build/en/33339/](https://vllm--33339.org.readthedocs.build/en/33339/)

