---
title: '19280'
pageLayout: default
tags:
  - pr-reviews
  - feat-
  - sig-
slug: thoughts/inference/vllm-prs/19280
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/19280.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#19280](https://github.com/aarnphm/aarnphm.github.io/issues/19280) \[V1\]\[Kernel\] Flashinfer HND KV cache layout

[vllm-project/vllm#19280](https://github.com/vllm-project/vllm/pull/19280) · NickLucche · MERGED · created 2025-06-06T12:32:05Z · merged 2025-06-17T13:09:22Z

Follow up PR to [vllm-project/vllm#18775](https://github.com/vllm-project/vllm/pull/18775), again porting over functionality from V0 (ref [vllm-project/vllm#16605](https://github.com/vllm-project/vllm/pull/16605)).

This PR will enable the use of FlashInfer with a HND cache layout in V1.
Among the most immediate benefits, this PR is a prerequisite to enabling heterogeneous TP support for disaggregated prefill-decode setup, optimizing the layout for xfers.

Test with:



```
FLASHINFER_KV_CACHE_LAYOUT=HND VLLM_ATTENTION_BACKEND=FLASHINFER vllm serve Qwen/Qwen3-0.6B
```

A simple benchmark:



```
# FLASHINFER_KV_CACHE_LAYOUT=HND  VLLM_ATTENTION_BACKEND=FLASHINFER vllm serve Qwen/Qwen3-14B --disable-log-requests

NHD

============ Serving Benchmark Result ============
Successful requests:                     982
Benchmark duration (s):                  94.63
Total input tokens:                      1745235
Total generated tokens:                  125696
Request throughput (req/s):              10.38
Output token throughput (tok/s):         1328.26
Total Token throughput (tok/s):          19770.61
---------------Time to First Token----------------
Mean TTFT (ms):                          45294.13
Median TTFT (ms):                        46790.26
P99 TTFT (ms):                           91790.54
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          106.51
Median TPOT (ms):                        113.65
P99 TPOT (ms):                           131.63
---------------Inter-token Latency----------------
Mean ITL (ms):                           106.51
Median ITL (ms):                         30.62
P99 ITL (ms):                            378.72
==================================================

HDN (this PR)

============ Serving Benchmark Result ============
Successful requests:                     982
Benchmark duration (s):                  95.03
Total input tokens:                      1745219
Total generated tokens:                  125696
Request throughput (req/s):              10.33
Output token throughput (tok/s):         1322.67
Total Token throughput (tok/s):          19687.22
---------------Time to First Token----------------
Mean TTFT (ms):                          45028.82
Median TTFT (ms):                        46518.14
P99 TTFT (ms):                           92111.68
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          106.67
Median TPOT (ms):                        111.74
P99 TPOT (ms):                           132.01
---------------Inter-token Latency----------------
Mean ITL (ms):                           106.67
Median ITL (ms):                         30.41
P99 ITL (ms):                            378.99
==================================================
```

cc @mgoin

---

## comments

**github-actions** (2025-06-06T12:33:21Z):

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at <https://slack.vllm.ai> to discuss your PR in <ref slug="tags/pr-reviews">, coordinate on features in <ref slug="tags/feat-"> channels, or join special interest groups in <ref slug="tags/sig-"> channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run `fastcheck` CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your `fastcheck` build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping `simon-mo` or `khluu` to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add `ready` label to the PR or enable auto-merge.

🚀

**robertgshaw2-redhat** (2025-06-06T17:40:50Z):

Did you do any performance tests to see how this impacts E2E performance for non-PD setups?

I think a throughput benchmark is warranted at least

**NickLucche** (2025-06-06T17:53:07Z):

I compared a few models (llama3/qwen3), didn’t notice a huge impact tbh. I’ve added a simple H100 benchmark in the description.

I should also probably tag @wenscarl to elaborate on HND for flashinfer.

**mergify** (2025-06-09T16:38:12Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @NickLucche.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**NickLucche** (2025-06-10T09:12:33Z):

I’ve refactored the flashinfer kv layout getter into a shared util.
I am using that method in flashattn method too so I can get rid of the annoying duplicated logs I am seeing on main all in one:



```
INFO 06-10 08:33:36 [utils.py:105] NixlConnector detected. Setting KV cache layout to HND for better xfer performance.
INFO 06-10 08:33:36 [utils.py:105] NixlConnector detected. Setting KV cache layout to HND for better xfer performance.
INFO 06-10 08:33:36 [utils.py:105] NixlConnector detected. Setting KV cache layout to HND for better xfer performance.
INFO 06-10 08:33:36 [utils.py:105] NixlConnector detected. Setting KV cache layout to HND for better xfer performance.
INFO 06-10 08:33:36 [utils.py:105] NixlConnector detected. Setting KV cache layout to HND for better xfer performance.
INFO 06-10 08:33:36 [utils.py:105] NixlConnector detected. Setting KV cache layout to HND for better xfer performance.
INFO 06-10 08:33:36 [utils.py:105] NixlConnector detected. Setting KV cache layout to HND for better xfer performance.
```

**NickLucche** (2025-06-12T09:02:40Z):

cc @LucasWilkinson

**mergify** (2025-06-13T21:49:20Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @NickLucche.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**NickLucche** (2025-06-14T18:29:09Z):

Thanks for the review! I’ve rebased the PR, let me know what you think.

