---
title: '45330'
pageLayout: default
slug: thoughts/inference/vllm-prs/45330
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/45330.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#45330](https://github.com/aarnphm/aarnphm.github.io/issues/45330) \[Bug\]: \[NIXL\] Hetero TP assertion fails when tp > num_kv_heads (GQA replication)

[vllm-project/vllm#45330](https://github.com/vllm-project/vllm/issues/45330) · kannakAWS · CLOSED · created 2026-06-11T22:57:32Z

### Your current environment

<details>
<summary>The output of <code>python collect_env.py</code></summary>

text
vLLM version: 0.21.0
Python version: 3.12.11
OS: Linux 6.12.58-82.121.amzn2023.x86_64 x86_64

</details>

### Describe the bug

\_validate_remote_agent_handshake in NixlConnectorWorker (worker.py:\~1498) asserts:

python
assert remote_block_len == (self.block_len_per_layer\[0\] \* tp_ratio) // block_size_ratio

This assumes block_len scales linearly with 1/tp_size. However, when tp_size > num_kv_heads, GQA replication caps num_kv_heads_per_rank at 1, so block_len stops scaling. Both sides end up with the same block_len despite different TP sizes, causing the assertion to fail.

### Steps to reproduce

Run disaggregated inference with:

- Model: Any model with few KV heads (e.g., meta-llama/Llama-3.2-1B-Instruct — 8 KV heads)
- Prefill: —tensor-parallel-size 8
- Decode: —tensor-parallel-size 16
- KV connector: NixlConnector

### Error

AssertionError: Remote P worker KV layer cache must be of shape \[2, N, local_kv_heads\*tp_ratio, page_size, head_dim\] and same dtype.

### Configurations tested (model with 8 KV heads)

| Config   | D_TP > kv_heads? | P_TP > kv_heads? | Result |
| -------- | ---------------- | ---------------- | ------ |
| tp2→tp4  | No               | No               | PASS   |
| tp4→tp8  | No (edge: 8=8)   | No               | PASS   |
| tp8→tp16 | Yes (16>8)       | No (edge)        | FAIL   |
| tp4→tp16 | Yes (16>8)       | No               | FAIL   |
| tp16→tp8 | No (edge)        | Yes (16>8)       | FAIL   |

### Root cause

When tp > num_kv_heads, num_kv_heads_per_rank = max(1, num_kv_heads // tp) = 1 (GQA replication). Block_len is proportional to num_kv_heads_per_rank, so it caps at
a single head’s worth. The assertion expects remote_block_len = local_block_len × tp_ratio but both sides report the same block_len due to this cap.

### Suggested fix

The validation should account for GQA replication. For example, skip the strict proportionality check when max(local_tp, remote_tp) > total_num_kv_heads:



```
if tp_ratio > 0:
    expected = (self.block_len_per_layer[0] * tp_ratio) // block_size_ratio
    if remote_block_len != expected:
        # Allow when GQA replication caps block_len on either side
        total_kv_heads = self.transfer_topo.total_num_kv_heads
        if max(self.transfer_topo.tp_size, remote_tp_size) > total_kv_heads:
            pass  # block_len doesn't scale beyond kv_head count
        else:
            assert False, (
                "Remote P worker KV layer cache must be of shape [2, N,"
                " local_kv_heads*tp_ratio, page_size, head_dim] and same dtype."
            )

```

### Before submitting a new issue…

- [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lots of frequently asked questions.

---

## comments

**ZhanqiuHu** (2026-06-12T17:59:58Z):

Do you have a model and hardware setup that can test GQA replication end-to-end?

**kannakAWS** (2026-06-16T20:06:39Z):

> Do you have a model and hardware setup that can test GQA replication end-to-end?

Hardware: 8x NVIDIA A100-SXM4-40GB (p4d.24xlarge equivalent)
Model: TinyLlama/TinyLlama-1.1B-Chat-v1.0 (32 attention heads, 4 KV heads)
Config: Prefill TP=2, Decode TP=8 (decode_tp=8 > num_kv_heads=4 triggers GQA replication)

**ZhanqiuHu** (2026-06-16T20:13:58Z):

Cool! Homogeneous + Heterogeneous TP with replication is already handled for hybrid mamba models like Nemotron and Qwen3.5+, so the replication handling logic should already be there (might be guarded by has mamba check). We intentionally keep the assertion for pure attention models because low `num_kv_heads` is relatively rare and we don’t have an e2e test case to verify that. If you could help verifying replication on your end, that would be great!

**waynehacking8** (2026-06-17T14:23:16Z):

For the e2e verification @ZhanqiuHu asked for — proving the _replicated KV data_ transfers correctly, not just that the handshake validates (the unit tests in the linked PRs check handshake math against a mocked `NixlWrapper`) — I’ve drafted a thin wrapper over the existing `nixl_integration` accuracy harness. It runs a real disaggregated prefill→decode through `NixlConnector` with `decode_tp > num_key_value_heads` and asserts the decode output matches the non-disaggregated baseline; replication-corrupted KV would make decode diverge, so a pass is genuine data-correctness evidence.



```bash
#!/bin/bash
# tests/v1/kv_connector/nixl_integration/run_gqa_replication_accuracy_test.sh
# E2E accuracy test for GQA KV replication over NIXL (heterogeneous TP).
# Verifies issue #45330: when DECODER_TP_SIZE > num_key_value_heads, GQA
# replication caps per-rank KV heads at 1, so prefill and decode report the
# same block_len despite tp_ratio > 1. Drives a real disaggregated
# prefill->decode transfer through NixlConnector and asserts decode output
# matches the non-disaggregated baseline.
#
# Reproduces the reporter's config: TinyLlama-1.1B (4 KV heads),
# PREFILLER_TP_SIZE=2, DECODER_TP_SIZE=8 (8 > 4 triggers replication).
# Requires enough GPUs for concurrent P(TP=2)+D(TP=8). Invariant under test:
# DECODER_TP_SIZE > num_key_value_heads.
set -xe
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P)"
MODEL_NAMES=${MODEL_NAMES:-"TinyLlama/TinyLlama-1.1B-Chat-v1.0"} \
PREFILLER_TP_SIZE=${PREFILLER_TP_SIZE:-2} \
DECODER_TP_SIZE=${DECODER_TP_SIZE:-8} \
DECODER_KV_LAYOUT=${DECODER_KV_LAYOUT:-"HND"} \
  exec "${SCRIPT_DIR}/run_accuracy_test.sh" "$@"
```

I don’t have a multi-GPU box to run it myself, but @kannakAWS has 8×A100 and it should fit the existing multi-GPU NIXL CI. Happy to open it as a small test-only PR so the landing fix ([#45879](https://github.com/aarnphm/aarnphm.github.io/issues/45879) / [#45860](https://github.com/aarnphm/aarnphm.github.io/issues/45860)) can be gated on it — let me know the preferred path.

**kannakAWS** (2026-06-17T20:33:29Z):

E2E verification complete - GQA replication works correctly with the fix applied.

Setup:

- Hardware: 8× A100-SXM4-40GB
- Model: TinyLlama/TinyLlama-1.1B-Chat-v1.0 (4 KV heads, 32 attn heads)
- Config: Prefill TP=2 → Decode TP=8 (tp8 > 4 kv_heads triggers GQA replication)
- vLLM 0.21.0 with remote_heads // local_heads fix applied to \_validate_remote_agent_handshake
- Disaggregated via NIXL + toy_proxy_server.py

Without fix:
`AssertionError: Remote P worker KV layer cache must be of shape [2, N, local_kv_heads*tp_ratio, page_size, head_dim] and same dtype.
`

With fix, e2e request through disaggregated pipeline:



```
curl http://localhost:8300/v1/completions -d '{"model":"TinyLlama/TinyLlama-1.1B-Chat-v1.0","prompt":"The capital of France is","max_tokens":10}'

{"choices":[{"text":" Paris Business language in Stock market has a significant role","finish_reason":"length"}]}

```

Handshake passes, KV transfers via NIXL, decode generates coherent output starting with “Paris”. Confirms data correctness - corrupted KV under replication would produce garbage.

@waynehacking8 your test wrapper script works as designed on 8×A100. Happy to run additional configs if needed.

Validated against the same fix approach used in [#45879](https://github.com/aarnphm/aarnphm.github.io/issues/45879) (@Oseltamivir) and [#45860](https://github.com/aarnphm/aarnphm.github.io/issues/45860) (mine) - all three PRs use remote_heads // local_heads instead of raw tp_ratio. LGTM on [#45879](https://github.com/aarnphm/aarnphm.github.io/issues/45879) which has comprehensive testing. Happy to see it land.

