---
title: '54687'
pageLayout: default
slug: thoughts/inference/vllm-prs/54687
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/54687.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#54687](https://github.com/aarnphm/aarnphm.github.io/issues/54687) \[Kernel\] Reuse Qwen4Exp HC combine-norm for MTP input

[vllm-project/vllm#54687](https://github.com/vllm-project/vllm/pull/54687) · gcanlin · MERGED · created 2026-09-01T03:29:00Z · merged 2026-09-04T09:57:56Z

## Summary

- Reuse the existing Qwen4Exp HC combine kernels for MTP input instead of maintaining an MTP-only add-and-norm kernel.
- Treat `injection_logits=None` as unit injection. Triton specializes directly on the optional `inj_ptr`, so the MTP path performs no injection load or sigmoid.
- Pass `fc_embedding` into the normal decoder as a pending HC block output. The existing combine-norm kernel then fuses the BF16 broadcast add and grouped Gemma RMSNorm.
- Generalize both HC combine variants to the same optional-injection contract, and remove `_forward_mtp_layer` plus the duplicated MTP kernel.

This does not duplicate [#54560](https://github.com/aarnphm/aarnphm.github.io/issues/54560), which targets the general Qwen4Exp `M=1` SM90 decode path; this change targets the MTP entry residual add and grouped RMSNorm for arbitrary token counts.

## Data flow



```text
target / feedback hidden                 new token
          |                                  |
 pre-norm + fc_hidden                embed + pre-norm
          |                                  |
 residual [T, HC, H]               fc_embedding [T, H]
          |                                  |
          |                         pending block output
          |                         injection = None
          +------------------+---------------+
                             |
                 Qwen4ExpDecoderLayer.forward
                             |
                  attn_hc.combine_and_mix
                             |
             hc_combine_norm[inj_ptr=None]
                    /                    \
       residual + embedding          grouped RMSNorm
                    \                    /
                     +--------+----------+
                              |
                  normal HC mix and decoder
                              |
                     Attention -> MLP
```

## Final kernel specialization

Following review, the final implementation keeps the optional-pointer design while minimizing special-case code:

- Residual and block loads are issued before FP32 conversion and arithmetic.
- Gated and unit-injection paths share the same two-dimensional tiled layout and RMSNorm reduction.
- Both paths use one flattened token-major launch grid, keeping the streams for each token adjacent without an injection-specific grid branch.
- The only unit-injection-specific scheduling is issuing the norm-weight load before the reduction; the gated path keeps the load after the reduction to limit its live range.

## Correctness



```bash
CUDA_VISIBLE_DEVICES=4 python3 -m pytest -q \
  tests/models/qwen4_exp/test_hc_ops.py
```

Result: `8 passed`. The tests cover gated and unit-injection combine paths, including 1, 17, and 2048 tokens.

## Kernel microbenchmark

NVIDIA B200, CUDA 13.0, PyTorch 2.13.0+cu130, BF16, `H=2560`, `HC=4`; CUPTI medians with CUDA graphs and cold L2. All baseline and current callables are captured with `torch.compile(fullgraph=True)`, and both materialized BF16 outputs are checked before timing.

The without-injection baseline is main’s separate broadcast add followed by grouped Gemma RMSNorm. The with-injection baseline is main’s existing injected HC combine-norm path. Thus each optional-pointer specialization is compared with the corresponding main compiled behavior.

### Without injection

| Tokens | Main compiled (us) | Current (us) | Speedup vs main |
| -----: | -----------------: | -----------: | --------------: |
|      1 |              3.680 |        2.400 |          1.533x |
|      2 |              4.096 |        2.432 |          1.684x |
|     16 |              4.672 |        2.560 |          1.825x |
|     64 |              6.848 |        2.975 |          2.302x |
|    256 |             15.713 |        4.672 |          3.363x |
|   2048 |             99.968 |       22.175 |          4.508x |
|   8192 |            393.344 |       81.216 |          4.843x |
|  32768 |           1560.477 |      323.968 |          4.817x |

### With injection

| Tokens | Main compiled (us) | Current (us) | Speedup vs main |
| -----: | -----------------: | -----------: | --------------: |
|      1 |              2.880 |        2.848 |          1.011x |
|      2 |              2.944 |        2.943 |          1.000x |
|     16 |              2.976 |        2.945 |          1.011x |
|     64 |              3.360 |        3.312 |          1.014x |
|    256 |              5.343 |        5.344 |          1.000x |
|   2048 |             25.024 |       23.872 |          1.048x |
|   8192 |            102.400 |       85.152 |          1.203x |
|  32768 |            396.000 |      327.808 |          1.208x |

The injected path is unchanged within measurement noise at small token counts and improves by about 20% at 8192 and 32768 tokens. The unit-injection MTP path remains 1.53x-4.84x faster than the corresponding main compiled sequence.

## End-to-end benchmark

Pending final paired rerun on TP4 B200.

## Model evaluation

Full GSM8K, 1,319 examples, 5-shot, greedy decoding:



```bash
OPENAI_API_KEY=EMPTY PYTHONPATH=. .venv/bin/python -m lm_eval run \
  --model local-chat-completions \
  --model_args model=qwen3.8-flash-next,base_url=http://127.0.0.1:8102/v1/chat/completions,tokenizer_backend=None,tokenized_requests=False,num_concurrent=64,max_gen_toks=1024,max_length=32768,timeout=600 \
  --tasks gsm8k --num_fewshot 5 --apply_chat_template \
  --gen_kwargs temperature=0,max_gen_toks=1024,reasoning_effort=none \
  --seed 42 \
  --output_path /tmp/qwen-mtp-e2e-5k1k-v4/gsm8k-current
```

- Flexible exact match: **0.9666 +/- 0.0049**
- Strict exact match: **0.9659 +/- 0.0050**

Before:

<img width="2926" height="766" alt="image" src="https://github.com/user-attachments/assets/64931645-1793-4e2f-8176-2e93c64d1554" />

After:

<img width="3020" height="824" alt="image" src="https://github.com/user-attachments/assets/1642d364-8fdd-4389-948c-fdeb6709f250" />

## AI assistance

OpenAI Codex assisted with implementation, testing, benchmarking, and drafting. The human submitter reviewed the change and is responsible for it.

---

## comments

**gcanlin** (2026-09-02T05:47:04Z):

@gau-nernst This PR should be ready now. It’s a bit difficult to see the e2e improvement because of the unstable acceptance.

Would it break the readability? I’m thinking whether this fusion is necessary.

**github-actions** (2026-09-03T17:03:04Z):

✅ @gcanlin, CI is now available for this PR.

- `/ci run` starts upstream CI; `/amd-ci run` starts AMD CI only.
- `/ci retry` retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
- `/amd-ci retry` retries failed jobs in AMD CI for the current PR head. Use `/amd-ci run` when the current head has no AMD CI build.
- `/ci cancel` cancels scheduled or running CI builds for this PR branch; `/amd-ci cancel` does the same for AMD CI only.

<!-- vllm-ci-authorized -->

**ZJY0516** (2026-09-03T17:03:38Z):

/ci run

**github-actions** (2026-09-03T17:03:58Z):

✅ Triggered [Buildkite CI #87108](https://buildkite.com/vllm/ci/builds/87108) for commit `f183b46aa20d`.

**coderabbitai** (2026-09-04T08:05:32Z):

<!-- This is an auto-generated comment: summarize by coderabbit.ai -->

<!-- review_stack_entry_start -->

[![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/vllm-project/vllm/pull/54687)

<!-- review_stack_entry_end -->

<!-- recent_review_start -->

No actionable comments were generated in the recent review. 🎉

<details>
<summary>ℹ️ Recent review info</summary>

<details>
<summary>⚙️ Run configuration</summary>

**Configuration used**: Repository UI

**Review profile**: CHILL

**Plan**: Team

**Run ID**: `155aa845-570b-4ef4-b616-8b761b04344d`

</details>

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between [`bb363db`](https://github.com/aarnphm/aarnphm.github.io/commit/bb363db9a5ec2edc7b39e99b00af363a89d1fb81) and [`563a407`](https://github.com/aarnphm/aarnphm.github.io/commit/563a407632e336340ced9eceba569e4316c8a2ac).

</details>

<details>
<summary>📒 Files selected for processing (5)</summary>

- `tests/models/qwen4_exp/test_hc_ops.py`
- `vllm/models/qwen4_exp/nvidia/hyperconnection.py`
- `vllm/models/qwen4_exp/nvidia/model.py`
- `vllm/models/qwen4_exp/nvidia/mtp.py`
- `vllm/models/qwen4_exp/nvidia/ops/hc.py`

</details>

**Included review availability:** Your plan provides up to 10 included reviews per hour; 9 remain after this review.

</details>

---

<!-- recent_review_end -->

<!-- walkthrough_start -->

<details>
<summary>📝 Summary</summary>

<!-- This is an auto-generated comment: release notes by coderabbit.ai -->

## Summary by CodeRabbit

- **New Features**
  - Qwen4Exp hyperconnection operations now support combining outputs without an injection value, applying a unit-weight fallback across streams.
  - Qwen4Exp decoder processing now preserves and applies pending block outputs consistently, including embedding projections.

- **Bug Fixes**
  - Improved handling of missing injection data during normalized and non-normalized residual combination.

- **Tests**
  - Added coverage for unit-weight injection behavior across multiple token counts.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

## Walkthrough

### Changes

Qwen4Exp HC combine operations now accept `None` injection logits and apply unit-weight block output. MTP preserves projected embeddings as pending block output. Decoder layers detect pending state from block output alone. Tests cover regular and normalized combines.

**Qwen4Exp HC injection handling**

| Layer / File(s)                                                                                                                                                      | Summary                                                                                                                                                              |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Optional injection kernel path** <br> `vllm/models/qwen4_exp/nvidia/ops/hc.py`                                                                                     | HC kernels, wrappers, and fake implementations accept optional injection logits. Missing logits skip gating and use unit-weight combination.                         |
| **Pending state propagation** <br> `vllm/models/qwen4_exp/nvidia/hyperconnection.py`, `vllm/models/qwen4_exp/nvidia/mtp.py`, `vllm/models/qwen4_exp/nvidia/model.py` | Gated residual methods accept optional injections. MTP passes projected embeddings as pending block output. Decoder layers use block output to detect pending state. |
| **Unit injection validation** <br> `tests/models/qwen4_exp/test_hc_ops.py`                                                                                           | Tests verify unit-weight combines and normalized results for token counts of 1, 17, and 2048.                                                                        |

**Estimated code review effort:** 3 (Moderate) | \~20 minutes

<!-- final_review_risk_start -->

**Merge Risk:** _⚪ Minimal_ · up to `563a4`

<!-- final_review_risk_coverage:{"sourceCommitId":"563a407632e336340ced9eceba569e4316c8a2ac","coveredCommitId":"563a407632e336340ced9eceba569e4316c8a2ac","kind":"reviewed"} -->

This change routes MTP embedding residuals through the shared HC combine-and-normalize path, using unit injection when logits are absent. The covered outputs match the intended residual and normalization behavior, with no current merge-blocking risk identified.

<!-- final_review_risk_end -->

**Suggested reviewers:** `peakcrosser7`

</details>

<!-- walkthrough_end -->

<!-- pre_merge_checks_walkthrough_start -->

<details>
<summary>🚥 Pre-merge checks | ✅ 4 | ❌ 1</summary>

### ❌ Failed checks (1 warning)

|     Check name     | Status     | Explanation                                                                                                                                                                                | Resolution                                                                         |
| :----------------: | :--------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------- |
| Docstring Coverage | ⚠️ Warning | Docstring coverage is 6.25% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 16 functions across 5 files. | Write docstrings for the functions missing them to satisfy the coverage threshold. |

<details>
<summary>✅ Passed checks (4 passed)</summary>

|         Check name         | Status   | Explanation                                                                                                                                      |
| :------------------------: | :------- | :----------------------------------------------------------------------------------------------------------------------------------------------- |
|     Linked Issues check    | ✅ Passed | Check skipped because no linked issues were found for this pull request.                                                                         |
| Out of Scope Changes check | ✅ Passed | Check skipped because no linked issues were found for this pull request.                                                                         |
|         Title check        | ✅ Passed | The title clearly summarizes the main change: reusing the Qwen4Exp HC combine-norm kernel for MTP input.                                         |
|      Description check     | ✅ Passed | The description directly explains the kernel reuse, optional injection behavior, decoder data flow, testing, benchmarks, and evaluation results. |

</details>

</details>

<!-- pre_merge_checks_walkthrough_end -->

- [ ] <!-- {"checkboxId":"585bb3f6-faf5-4dbf-96d2-74e382adf19a"} --> Fix all pre-merge checks with AI

<!-- tips_start -->

---

Thanks for using [CodeRabbit](https://coderabbit.ai?utm_source=oss\&utm_medium=github\&utm_campaign=vllm-project/vllm\&utm_content=54687)! It’s free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

<details>
<summary>❤️ Share</summary>

- [X](https://twitter.com/intent/tweet?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A\&url=https%3A//coderabbit.ai)
- [Mastodon](https://mastodon.social/share?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A%20https%3A%2F%2Fcoderabbit.ai)
- [Reddit](https://www.reddit.com/submit?title=Great%20tool%20for%20code%20review%20-%20CodeRabbit\&text=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code.%20Check%20it%20out%3A%20https%3A//coderabbit.ai)
- [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcoderabbit.ai\&mini=true\&title=Great%20tool%20for%20code%20review%20-%20CodeRabbit\&summary=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code)

</details>

<sub>Comment `@coderabbitai help` to get the list of available commands.</sub>

<!-- tips_end -->

**gcanlin** (2026-09-04T08:13:32Z):

/ci run

**github-actions** (2026-09-04T08:13:50Z):

✅ Triggered [Buildkite CI #87221](https://buildkite.com/vllm/ci/builds/87221) for commit `42e6d6144322`.

