---
title: '30207'
pageLayout: default
slug: thoughts/inference/vllm-prs/30207
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/30207.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#30207](https://github.com/aarnphm/aarnphm.github.io/issues/30207) Enable Cross layers KV cache layout at NIXL Connector

[vllm-project/vllm#30207](https://github.com/vllm-project/vllm/pull/30207) · liranschour · MERGED · created 2025-12-07T13:17:01Z · merged 2026-01-22T10:12:58Z

<!-- markdownlint-disable -->

## Purpose

Enable NIXL Connector to us the new continuous cross layer KV cache layout described in RFC [#27742](https://github.com/vllm-project/vllm/issues/27742) and implemented in [#27743](https://github.com/vllm-project/vllm/pull/27743).
Demonstrate performance improvement of more the 2x in Tok/sec and TTFT due to dramatic reduction of fragmentation of transfer buffers.

Tested with P!=D with run_accuracy_test.sh P=1 D=2

Tested on H100 1P1D Qwen/Qwen3-0.6B:

| branch          | num reqs | input len | TTFT     | ITL   | tok/s     | Desc/transfer |
| --------------- | -------- | --------- | -------- | ----- | --------- | ------------- |
| main            | 1000     | 16        | 18756.42 | 5.35  | 5288.41   | 56            |
| kv_cross_layers | 1000     | 16        | 8494.44  | 8.74  | 8572.56   | 1             |
| main            | 128      | 1024      | 1660.84  | 9.40  | 37945.20  | 3528          |
| kv_cross_layers | 128      | 1024      | 686.98   | 9.26  | 55418.76  | 1             |
| main            | 128      | 10240     | 11140.52 | 42.78 | 62339.74  | 34000         |
| kv_cross_layers | 128      | 10240     | 5226.71  | 14.41 | 117631.48 | 422           |

---

## comments

**mergify** (2025-12-07T13:21:20Z):

Hi @liranschour, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

**mergify** (2025-12-08T11:08:03Z):

Hi @liranschour, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

**mergify** (2025-12-11T09:08:38Z):

Hi @liranschour, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

> \[!tip\]
>
> <details>
> <summary>Is <code>mypy</code> or <code>markdownlint</code> failing?</summary>
> <br/>
> <code>mypy</code> and <code>markdownlint</code> are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:
>
>
>
> ```bash
> # For mypy (substitute "3.10" with the failing version if needed)
> pre-commit run --hook-stage manual mypy-3.10
> # For markdownlint
> pre-commit run --hook-stage manual markdownlint
> ```
>
> </details>

**mergify** (2025-12-11T13:30:57Z):

Hi @liranschour, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

> \[!tip\]
>
> <details>
> <summary>Is <code>mypy</code> or <code>markdownlint</code> failing?</summary>
> <br/>
> <code>mypy</code> and <code>markdownlint</code> are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:
>
>
>
> ```bash
> # For mypy (substitute "3.10" with the failing version if needed)
> pre-commit run --hook-stage manual mypy-3.10
> # For markdownlint
> pre-commit run --hook-stage manual markdownlint
> ```
>
> </details>

**mergify** (2025-12-14T14:31:30Z):

Hi @liranschour, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

> \[!tip\]
>
> <details>
> <summary>Is <code>mypy</code> or <code>markdownlint</code> failing?</summary>
> <br/>
> <code>mypy</code> and <code>markdownlint</code> are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:
>
>
>
> ```bash
> # For mypy (substitute "3.10" with the failing version if needed)
> pre-commit run --hook-stage manual mypy-3.10
> # For markdownlint
> pre-commit run --hook-stage manual markdownlint
> ```
>
> </details>

**mergify** (2025-12-17T10:58:57Z):

Hi @liranschour, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

> \[!tip\]
>
> <details>
> <summary>Is <code>mypy</code> or <code>markdownlint</code> failing?</summary>
> <br/>
> <code>mypy</code> and <code>markdownlint</code> are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:
>
>
>
> ```bash
> # For mypy (substitute "3.10" with the failing version if needed)
> pre-commit run --hook-stage manual mypy-3.10
> # For markdownlint
> pre-commit run --hook-stage manual markdownlint
> ```
>
> </details>

**mergify** (2025-12-17T12:50:40Z):

Hi @liranschour, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

> \[!tip\]
>
> <details>
> <summary>Is <code>mypy</code> or <code>markdownlint</code> failing?</summary>
> <br/>
> <code>mypy</code> and <code>markdownlint</code> are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:
>
>
>
> ```bash
> # For mypy (substitute "3.10" with the failing version if needed)
> pre-commit run --hook-stage manual mypy-3.10
> # For markdownlint
> pre-commit run --hook-stage manual markdownlint
> ```
>
> </details>

**mergify** (2025-12-18T12:04:21Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @liranschour.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**mergify** (2026-01-12T16:13:09Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @liranschour.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**orozery** (2026-01-18T07:35:38Z):

@liranschour `test_nixl_connector.py` unit test passes, but I think none of the tests there actually test the cross-layers case.
Can we add such test?

**mergify** (2026-01-21T12:21:59Z):

Documentation preview: [https://vllm—30207.org.readthedocs.build/en/30207/](https://vllm--30207.org.readthedocs.build/en/30207/)

**mergify** (2026-01-21T12:31:53Z):

Hi @liranschour, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

> \[!tip\]
>
> <details>
> <summary>Is <code>mypy</code> or <code>markdownlint</code> failing?</summary>
> <br/>
> <code>mypy</code> and <code>markdownlint</code> are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:
>
>
>
> ```bash
> # For mypy (substitute "3.10" with the failing version if needed)
> pre-commit run --hook-stage manual mypy-3.10
> # For markdownlint
> pre-commit run --hook-stage manual markdownlint
> ```
>
> </details>

**mergify** (2026-01-21T13:06:26Z):

Hi @liranschour, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

> \[!tip\]
>
> <details>
> <summary>Is <code>mypy</code> or <code>markdownlint</code> failing?</summary>
> <br/>
> <code>mypy</code> and <code>markdownlint</code> are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:
>
>
>
> ```bash
> # For mypy (substitute "3.10" with the failing version if needed)
> pre-commit run --hook-stage manual mypy-3.10
> # For markdownlint
> pre-commit run --hook-stage manual markdownlint
> ```
>
> </details>

**mergify** (2026-01-21T13:19:06Z):

Hi @liranschour, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

> \[!tip\]
>
> <details>
> <summary>Is <code>mypy</code> or <code>markdownlint</code> failing?</summary>
> <br/>
> <code>mypy</code> and <code>markdownlint</code> are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:
>
>
>
> ```bash
> # For mypy (substitute "3.10" with the failing version if needed)
> pre-commit run --hook-stage manual mypy-3.10
> # For markdownlint
> pre-commit run --hook-stage manual markdownlint
> ```
>
> </details>

**mergify** (2026-01-21T13:32:31Z):

Hi @liranschour, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

> \[!tip\]
>
> <details>
> <summary>Is <code>mypy</code> or <code>markdownlint</code> failing?</summary>
> <br/>
> <code>mypy</code> and <code>markdownlint</code> are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:
>
>
>
> ```bash
> # For mypy (substitute "3.10" with the failing version if needed)
> pre-commit run --hook-stage manual mypy-3.10
> # For markdownlint
> pre-commit run --hook-stage manual markdownlint
> ```
>
> </details>

**mergify** (2026-01-21T14:18:30Z):

Hi @liranschour, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

> \[!tip\]
>
> <details>
> <summary>Is <code>mypy</code> or <code>markdownlint</code> failing?</summary>
> <br/>
> <code>mypy</code> and <code>markdownlint</code> are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:
>
>
>
> ```bash
> # For mypy (substitute "3.10" with the failing version if needed)
> pre-commit run --hook-stage manual mypy-3.10
> # For markdownlint
> pre-commit run --hook-stage manual markdownlint
> ```
>
> </details>

