---
title: '58957'
pageLayout: default
slug: thoughts/inference/vllm-prs/58957
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/58957.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#58957](https://github.com/aarnphm/aarnphm.github.io/issues/58957) \[Perf\]\[Qwen4Exp\] Fuse HC down projection and SiLU on NVIDIA

[vllm-project/vllm#58957](https://github.com/vllm-project/vllm/pull/58957) · gau-nernst · MERGED · created 2026-09-28T02:29:32Z · merged 2026-09-28T10:51:38Z

## Purpose

Fuse Qwen4Exp’s BF16 HC down projection and SiLU for NVIDIA decode batches up to 48 tokens, with the existing unfused path for larger batches. This is separate from [#58706](https://github.com/aarnphm/aarnphm.github.io/issues/58706) (ROCm) and [#53909](https://github.com/aarnphm/aarnphm.github.io/issues/53909) (other HC operations).

The CuTe DSL kernel adapts vLLM’s `ll_bf16` dot-product and split-K GEMM paths. It applies SiLU in the output epilogue, preserves the `ll_bf16` BF16 rounding boundary, and skips the 12 unused padding columns.

## Microbenchmarks

GB300, FlashInfer CUPTI with CUDA graph and cold L2; median latency in µs. Unfused is the Qwen4Exp serving GEMM (`shape_dynamic_skinny_gemm` at M=1/2/4/8, otherwise `F.linear`) + `hc_silu`.

| Tokens (M) |  Unfused |     Fused |   Speedup |
| ---------: | -------: | --------: | --------: |
|          1 |     5.86 |      4.16 |     1.41× |
|          2 |     5.95 |      4.42 |     1.35× |
|          4 |     6.75 |      6.02 |     1.12× |
|          8 |     8.29 |      5.54 |     1.50× |
|         16 |    10.02 |      5.66 |     1.77× |
|         32 |     9.76 |      6.27 |     1.56× |
|         48 |     9.76 |      6.94 |     1.41× |
|         56 |    10.91 |     10.72 |     1.02× |
|         57 |    11.04 |     11.11 |     0.99× |
|     **64** | **9.50** | **11.14** | **0.85×** |

All M=1–48 were measured and won. The crossover is around M=57 (near timing noise); M=64 clearly loses. Rows above M=48 force the fused kernel for comparison, while serving dispatch uses the unfused fallback there.

## E2E perf

Qwen3.8-Flash-Next, TP4 + MTP3, 8k input / 1k output, SPEED-Bench; median time per output token (TPOT). Latest `main` (`a2be4d3cc7`) and the rebased branch (`d2ab3e74ee`) ran sequentially on the same GB300 node:

| Concurrency | Main (ms) | Branch (ms) | Change |
| ----------: | --------: | ----------: | -----: |
|           1 |     4.574 |       4.444 | −2.84% |
|           4 |     6.259 |       6.130 | −2.06% |
|          16 |    10.082 |       9.681 | −3.98% |
|          64 |    19.957 |      19.883 | −0.37% |

Serving dispatch uses the unfused fallback above 48 tokens.

## Test plan

- `.venv/bin/python -m pytest tests/models/qwen4_exp/test_hc_ops.py -q`: 13 passed on GB300; `pre-commit run --files` on all changed files passed.

E2E accuracy A/B, TP4 + MTP3:

| Task     | Main (strict / flexible) | Branch (strict / flexible) |
| -------- | -----------------------: | -------------------------: |
| GSM8K    |          0.9674 / 0.9666 |            0.9663 / 0.9655 |
| MMMU Pro |          0.7636 / 0.7636 |            0.7642 / 0.7636 |

The E2E runs used an earlier branch revision, before the integration cleanup. AI assistance: Kimi Code and Codex helped develop and review the change.

---

## comments

**gau-nernst** (2026-09-28T09:22:32Z):

/ci run

**github-actions** (2026-09-28T09:23:56Z):

✅ Triggered [Buildkite CI #91564](https://buildkite.com/vllm/ci/builds/91564) for commit `49a4ee1c04fd`.

