---
title: '26556'
pageLayout: default
slug: thoughts/inference/vllm-prs/26556
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/26556.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#26556](https://github.com/aarnphm/aarnphm.github.io/issues/26556) \[NIXL\]\[HeteroTP\]Enable KV transfer from HND prefill to NHD decode

[vllm-project/vllm#26556](https://github.com/vllm-project/vllm/pull/26556) · xuechendi · MERGED · created 2025-10-10T03:10:39Z · merged 2025-10-14T09:46:05Z

<!-- markdownlint-disable -->

## Purpose

Current codes only support prefill and decode with same KV cache layout.
In this PR, would like to enable for new option that when prefill is HND, and receiver is HND.
In that case, nix&#x6C;_&#x72;ead can still copy data in big chunk (num_kv_head \* t&#x70;_&#x72;atio \* block_size \* head_size), and meanwhile, decode attn is not limited to HND only attention.

## Test Plan

Tested with vllm/tests/v1/kv_connector/nixl_integration/run_accuracy_test.sh

prefill:



```
    BASE_CMD="CUDA_VISIBLE_DEVICES=3" \
    UCX_NET_DEVICES=all \
    VLLM_NIXL_SIDE_CHANNEL_PORT=$SIDE_CHANNEL_PORT \
    vllm serve $model_name \
    --port $PORT \
    --enforce-eager \
    --block-size 128 \
    --gpu-memory-utilization 0.2 \
    --tensor-parallel-size 1 \
    --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}'"
```

decode:



```
BASE_CMD="CUDA_VISIBLE_DEVICES=4, 5" \
    UCX_NET_DEVICES=all \
    VLLM_KV_CACHE_LAYOUT=NHD \
    VLLM_NIXL_SIDE_CHANNEL_PORT=$SIDE_CHANNEL_PORT \
    vllm serve $model_name \
    --port $PORT \
    --enforce-eager \
    --block-size 128 \
    --gpu-memory-utilization 0.2 \
    --tensor-parallel-size 2 \
    --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}'"
```

‘gsm8k’ test:
‘model’: Qwen/Qwen3-0.6B
‘exact_match,strict-match’: **0.41849**, ‘exact_match_stderr,strict-match’: 0.013588287284030885,
‘exact_match,flexible-extract’: **0.41622**, ‘exact_match_stderr,flexible-extract’: 0.013577788334652657

Expected: 0.41

## Test Result

---

<details>
<summary> Essential Elements of an Effective PR Description Checklist </summary>

- [ ] The purpose of the PR, such as “Fix some issue (link existing issues this PR will resolve)”.
- [ ] The test plan, such as providing test command.
- [ ] The test results, such as pasting the results comparison before and after, or e2e results
- [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.
- [ ] (Optional) Release notes update. If your change is user facing, please update the release notes draft in the [Google Doc](https://docs.google.com/document/d/1YyVqrgX4gHTtrstbq8oWUImOyPCKSGnJ7xtTpmXzlRs/edit?tab=t.0).

</details>

---

## comments

**xuechendi** (2025-10-10T03:11:18Z):

@NickLucche , may you help to review this PR

**xuechendi** (2025-10-10T15:05:18Z):

@NickLucche , I have resolved all comments.
BTW, I also plan to enable for prefill and decode are with different block_size, ex: prefill uses block_size=16 and decode uses 64. <span>&rArr;</span> Which will be good to do all reshape in one for loop.

I want to keep PR clean to one task. So I’ll do that in another PR. But for this one, do you think I should rename the function name as ‘permute_reshape_device_kv’?

**xuechendi** (2025-10-10T16:16:40Z):

> Hey @xuechendi thanks a lot for the work, this is
>
> > an interesting idea!
>
> Left some comments, ptal.
>
> Overall, I think we can allow D to be NHD. However, as we’re not measuring the overhead these extra copies are introducing, I would like to have `--enable_permute_local_kv` (or similar) as a flag to set on vllm in order to allow this experimental feature in. We can turn it on by default on Intel hw. What I would like to avoid is a silent drop in speed on misconfigured system until our CI is solid enough to test out multiple cases (eg see MLA comment I left, but also FI or tp_size>kv_heads case..). Let me know if that works for you.
>
> Speaking of CI, would you mind setting up a test to showcase your feature?

@NickLucche ,

regarding to the `--enable_permute_local_kv`, the change does not affect existing working path (when sender and receiver with same KV layout ), the change only takes affect for `HND` to `NHD` case, which goes assert currently.
For your suggestion, do you think we should still keep assert as default and only with `--enable_permute_local_kv` enabled, we make `HND` to `NHD` work?

For MLA validation, I did validation with deepseek-v2-lite-chat, prefill TP1 + HND, decode TP1 + NHD
‘gsm8k’ <span>&rArr;</span> which gets expected accuracy
‘exact_match,strict-match’: **0.6484375**, ‘exact_match_stderr,strict-match’: 0.029899590697818237,
‘exact_match,flexible-extract’: **0.65234375**, ‘exact_match_stderr,flexible-extract’: 0.029822440677853123

Performance wisely, here is current measurement:
<span>&rArr;</span> seems permute overhead is not significant. Not sure why for TP1 case, with permute actually gets better e2e time.

| A100 Gsm8k E2e time   | Prefill TP1 + HND Decode TP1 + HND (existing) | Prefill TP1 + HND Decode TP1 + NHD (new feature from this PR) | Prefill TP1 + HND Decode TP2 + HND (existing) | Prefill TP1 + HND Decode TP2 + NHD (new feature from this PR) |
| --------------------- | --------------------------------------------- | ------------------------------------------------------------- | --------------------------------------------- | ------------------------------------------------------------- |
| Deepseek-v2-lite-chat | 61.037 s                                      | 60.309 s                                                      | 77.870 s                                      | 78.225 s                                                      |
| Qwen/Qwen3-0.6B       | 97.759 s                                      | 94.552 s                                                      | 101.737 s                                     | 104.726 s                                                     |

I also did a profiling to show the permute_device_kv overhead:
input lens is 2\*288 tokens = \~512 tokens
decoder side get_finished() took 194ms, permute_device_kv took 45ms.

<img width="1444" height="644" alt="image" src="https://github.com/user-attachments/assets/0a7509ff-4b32-4b61-b0e5-6665e2ffbd05" />

Below the original ‘HND’ <span>&rarr;</span> ‘HND’ profiling, behavior stays same as before
<img width="1437" height="815" alt="image" src="https://github.com/user-attachments/assets/135c5e20-8f05-421c-a1cd-374044d544b1" />

**xuechendi** (2025-10-10T16:29:08Z):

@NickLucche
I added one arg in run_accuracy_test.sh to test with this new feature
<img width="655" height="386" alt="image" src="https://github.com/user-attachments/assets/825397fe-b39c-452d-8925-a721b9b3dbdc" />

**NickLucche** (2025-10-13T08:13:38Z):

Thanks a lot for the work @xuechendi !

> For your suggestion, do you think we should still keep assert as default and only with —enable_permute_local_kv enabled, we make HND to NHD work?

Yep that’s what I was suggesting. Although I am happy with the way permutation is implemented, I still think the feature should be opt-in especially on cuda for max performance: we want the user to really be aware when giving up on layout parity, unless there’s no other alternative as for xpu, imo. At least when adding a new feature.

> decoder side get_finished() took 194ms, permute_device_kv took 45ms.

This could be sensible for bigger workloads (8/10k ISL) on deeper models, where P/D has its target use-case rn. Regardless, I think there’s ways to hide some of that overhead with async scheduling.

> I added one arg in run_accuracy_test.sh to test with this new feature

Thanks. As a last effort, would you mind adding one unit test in test_nixl_connector.py? Eg at least testing handhsake succeeds. Running all the sweep for lm_eval during dev will get take more and more time as we add more features.

**mergify** (2025-10-13T18:08:54Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @xuechendi.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**mergify** (2025-10-13T20:38:28Z):

Documentation preview: [https://vllm—26556.org.readthedocs.build/en/26556/](https://vllm--26556.org.readthedocs.build/en/26556/)

**xuechendi** (2025-10-13T21:05:51Z):

@NickLucche, resolved all comments

“—enable_permute_local_kv” configuration added: <https://github.com/vllm-project/vllm/pull/26556/commits/3082bbdbddd1dd80a86d27b5f089955b341403c0>

New UT added to test_nixl_connector.py: <https://github.com/vllm-project/vllm/pull/26556/commits/73dceb5080fda4b982d32cb54cb28e04354296a3>

updated run_accuracy_test.sh and nixl_connector_usage.md with new argument.

