---
title: '59533'
pageLayout: default
slug: thoughts/inference/vllm-prs/59533
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/59533.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#59533](https://github.com/aarnphm/aarnphm.github.io/issues/59533) \[Perf\]\[Qwen4Exp\] Merge QSA QKVG and indexer QK projections

[vllm-project/vllm#59533](https://github.com/vllm-project/vllm/pull/59533) · gau-nernst · MERGED · created 2026-10-01T03:19:18Z · merged 2026-10-05T07:03:30Z

## Overview

**Projection fusion.** Merge Qwen4Exp QSA’s QKVG and replicated indexer Q/K projections into **one GEMM**. LoRA keeps separate projections with standard linear dispatch.

**GEMM dispatch.** LL-GEMM for merged GEMM

- SM103 (GB300): TP4 shape, done by me
- SM100 (B200): TP4 contributed by @namgyu-youn
- SM90 (H200): TP4 contributed tuning for M=1/2/4
- SM121 (GB10): TP1 contributed by @stecasta
- Also prune obsolete merged-HC and separate-QSA plans;

**Prepare kernel.** Support strided merged outputs and widen row-offset arithmetic to i64, with **no noticeable performance difference** in GB300 checks (≤1% in repeat measurements).

## Validation

### Microbenchmark

One GB300, BF16, TP=4: `X=[M,2560]`, separate weights `[3584,2560]` + `[640,2560]`, merged weight `[4224,2560]`. Both arms use production `qwen4_exp_low_latency_gemm`: LL-GEMM at M=1/4, cuBLAS via standard linear at M>=16. The separate arm times both GEMMs without concatenation, using the original dispatch table from `9e6550bd05`; the reproduction restores that table for the baseline after pruning.

FlashInfer CUPTI, CUDA graphs, cold L2; median of three runs × 50 iterations, alternating arm order. Correctness checked against FP32 accumulation (`rtol=0.02`, `atol=0.01`). Torch `2.13.0+cu130`, baseline `9e6550bd05`.

|     M | Separate (µs) | Merged (µs) | Speedup |
| ----: | ------------: | ----------: | ------: |
|     1 |         9.857 |       6.624 |   1.49x |
|     4 |        11.024 |       7.616 |   1.45x |
|    16 |        13.472 |       8.032 |   1.68x |
|    64 |        13.664 |       8.224 |   1.66x |
|   256 |        13.584 |       9.216 |   1.47x |
|  1024 |        21.920 |      15.584 |   1.41x |
|  4096 |        51.809 |      51.648 |   1.00x |
| 16384 |       191.424 |     185.233 |   1.03x |

Largest spread between run medians: 5.5%. These measure projection GEMMs only; M=4096 is effectively neutral.

<details>
<summary>Microbenchmark reproduction</summary>

Run in a vLLM CUDA development venv with matching FlashInfer and `cupti-python` installed. Save the following script as `workspace/benchmark_qsa_projection_dispatch.py`, then run:



```bash
mkdir -p workspace
CUDA_VISIBLE_DEVICES=0 VLLM_BATCH_INVARIANT=0 .venv/bin/python workspace/benchmark_qsa_projection_dispatch.py
```



```python
import ast
import dataclasses
import json
import statistics
import subprocess
from pathlib import Path
from unittest.mock import patch

import torch
from flashinfer.testing import bench_gpu_time_with_cupti
from vllm.models.qwen4_exp.nvidia import low_latency_gemm as dispatch

M_VALUES = (1, 4, 16, 64, 256, 1024, 4096, 16384)
ROUNDS = 3
REPEAT_ITERS = 50
BASELINE_COMMIT = '9e6550bd058560bec5690b6c06474adf048a9b85'


def baseline_plans():
  source = subprocess.check_output(
    [
      'git',
      'show',
      f'{BASELINE_COMMIT}:vllm/models/qwen4_exp/nvidia/low_latency_gemm.py',
    ],
    text=True,
  )
  node = next(
    node
    for node in ast.parse(source).body
    if isinstance(node, ast.AnnAssign)
    and node.target.id == 'QWEN4_EXP_GEMM_PLANS'
  )
  namespace = {'SkinnyGemmConfig': dispatch.SkinnyGemmConfig}
  exec(
    compile(
      ast.Module(body=[node], type_ignores=[]), 'baseline-plans', 'exec'
    ),
    namespace,
  )
  return namespace['QWEN4_EXP_GEMM_PLANS']


def timing(fn, cold):
  samples = bench_gpu_time_with_cupti(
    fn,
    dry_run_iters=5,
    repeat_iters=REPEAT_ITERS,
    use_cuda_graph=True,
    cold_l2_cache=cold,
  )
  return statistics.median(samples) * 1000


def main():
  torch.manual_seed(42)
  weight = torch.randn(4224, 2560, device='cuda', dtype=torch.bfloat16) * 0.02
  qkv_weight, index_weight = weight.split((3584, 640), dim=0)
  real_skinny = dispatch.shape_dynamic_skinny_gemm
  records = []

  def spy(x, w, config):
    records.append((tuple(w.shape), dataclasses.asdict(config)))
    return real_skinny(x, w, config)

  results = {
    'gpu': torch.cuda.get_device_name(),
    'torch': torch.__version__,
    'baseline_commit': BASELINE_COMMIT,
    'candidate_commit': subprocess.check_output(
      ['git', 'rev-parse', 'HEAD'], text=True
    ).strip(),
    'dtype': 'bf16',
    'tp': 4,
    'k': 2560,
    'separate_n': [3584, 640],
    'merged_n': 4224,
    'm_values': M_VALUES,
    'rounds': ROUNDS,
    'iterations_per_round': REPEAT_ITERS,
    'cuda_graph': True,
    'results': [],
  }
  old_plans = baseline_plans()
  capability = dispatch.current_platform.get_device_capability()
  merged_plans = dispatch.QWEN4_EXP_GEMM_PLANS_BY_CAPABILITY.get(
    capability, {}
  )
  op = torch.ops.vllm.qwen4_exp_low_latency_gemm
  for m in M_VALUES:
    x = torch.randn(m, 2560, device='cuda', dtype=torch.bfloat16)

    def separate():
      with patch.dict(
        dispatch.QWEN4_EXP_GEMM_PLANS_BY_CAPABILITY, {capability: old_plans}
      ):
        return op(x, qkv_weight), op(x, index_weight)

    def merged():
      return op(x, weight)

    arms = {'separate': separate, 'merged': merged}
    row = {'m': m, 'dispatch': {}}
    for name, fn in arms.items():
      records.clear()
      with patch.object(
        dispatch, 'shape_dynamic_skinny_gemm', side_effect=spy
      ):
        output = fn()
      row['dispatch'][name] = list(records)
      sizes = (3584, 640) if name == 'separate' else (4224,)
      plans = old_plans if name == 'separate' else merged_plans
      expected = sum(m in plans.get((n, 2560), {}) for n in sizes)
      assert len(records) == expected, (m, name, expected, records)
      outputs = (
        output
        if isinstance(output, tuple)
        else output.split((3584, 640), dim=1)
      )
      for actual, w in zip(outputs, (qkv_weight, index_weight)):
        reference = torch.nn.functional.linear(x.float(), w.float()).bfloat16()
        torch.testing.assert_close(actual, reference, rtol=2e-2, atol=1e-2)
      for _ in range(10):
        fn()
    torch.cuda.synchronize()
    for cold in (True,):
      mode = 'cold' if cold else 'hot'
      times = {name: [] for name in arms}
      for run in range(ROUNDS):
        order = (
          ('separate', 'merged') if run % 2 == 0 else ('merged', 'separate')
        )
        for name in order:
          times[name].append(timing(arms[name], cold))
      row[mode] = {
        name: {
          'median_us': statistics.median(ts),
          'run_medians_us': ts,
          'tflops': 2 * m * 2560 * 4224 / (statistics.median(ts) * 1e6),
        }
        for name, ts in times.items()
      }
      row[mode]['speedup'] = (
        row[mode]['separate']['median_us'] / row[mode]['merged']['median_us']
      )
    results['results'].append(row)
    Path('workspace/qsa_projection_dispatch_results.json').write_text(
      json.dumps(results, indent=2) + '\n'
    )
    print(json.dumps(row), flush=True)


if __name__ == '__main__':
  main()
```

</details>

### Local correctness

**16 prepare tests passed before rebase; the focused LL-GEMM suite passed again after rebase: 7 passed, 102 architecture-specific skips.**



```bash
CUDA_VISIBLE_DEVICES=1 .venv/bin/python -m pytest tests/models/qwen4_exp/test_qsa_prepare.py -q --tb=short
CUDA_VISIBLE_DEVICES=1 .venv/bin/python -m pytest tests/kernels/test_bf16_skinny_gemm.py -k qwen4_exp -q --tb=short
```

Local checks also passed for BF16/FP8 projections and checkpoint loading across TP=1/2/4/8, LoRA fallback construction, and FP8 per-tensor/block scales. Sparse inputs crossing `2**31` elements verified the widened offsets. Another 36 checks verified standard-linear dispatch for pruned shapes. Capability-map selection and unsupported-capability fallback checks passed; CUDA projection checks at M=1/2/4/16 also passed after dispatch cleanup. Existing Buildkite suites cover QSA prepare/reference and LL-GEMM tests. The contributed H200 merged-GEMM plans passed local table and routing checks; H200 GPU correctness tests were skipped on this host.

### End-to-end performance

**TEP4 + MTP3, 4× GB300**, Qwen4Exp BF16. Baseline `aee8fe202a` (validation baseline) → feature `401b1249c0`, run sequentially on the same node with matched dependencies. Existing SPEED-Bench `throughput_8k` workload: inputs capped at 8192 tokens, 1024 output tokens, prefix caching off, five warmups, ten prompts per concurrency slot.

| Concurrency | Output tokens/s, base → feature | Change | Mean TTFT (ms), base → feature | Mean TPOT (ms), base → feature |
| ----------: | ------------------------------: | -----: | -----------------------------: | -----------------------------: |
|           1 |                   317.0 → 315.7 |  -0.4% |                  157.8 → 145.6 |                  3.003 → 3.028 |
|           4 |                   843.0 → 855.6 |  +1.5% |                  198.6 → 187.6 |                  4.358 → 4.322 |
|          16 |                 1883.4 → 1934.0 |  +2.7% |                  335.3 → 260.1 |                  7.980 → 7.788 |
|          64 |                 3566.3 → 3747.6 |  +5.1% |                  564.0 → 561.4 |                17.046 → 16.237 |

**0 failed requests** across 850 measured requests per arm. One sweep per arm; no repeated-run variability estimate. Throughput is effectively neutral at concurrency 1 and improves up to 5.1% at concurrency 64. MTP acceptance was 36.56% in both arms at concurrency 64.

### End-to-end accuracy

**Feature only, TEP4 + MTP3**, same BF16 model and node; prefix caching on. GSM8K: 5-shot, two epochs, temperature 0, thinking off, max 1024 tokens, concurrency 32. MMMU-Pro: thinking on, max 65536 tokens, concurrency 64.

| Task     | Responses | Flexible score | Strict score |
| -------- | --------: | -------------: | -----------: |
| GSM8K    |      2638 |         96.55% |       96.63% |
| MMMU-Pro |      1730 |         76.65% |       76.59% |

**0 request errors** for both tasks. Scores are close to historical results (\~96.7% GSM8K, 76.94% MMMU-Pro); no contemporaneous baseline accuracy run was requested.

<details>
<summary>End-to-end reproduction</summary>

Serving and performance commands below reproduce the tested configuration. Run each worktree with its independent venv on the same four GPUs, using the same BF16 checkpoint. Accuracy settings are listed above; the original evaluation harness is not included here.

Shared serving environment: `VLLM_USE_V2_MODEL_RUNNER=1`, `VLLM_ALLREDUCE_USE_FLASHINFER=1`. Torch `2.13.0+cu130`, Triton `3.7.1`, FlashInfer `0.7.0.post1`; Python 3.12.13. Source SHAs are recorded from git.



```bash
# Set this to the BF16 checkpoint used for both arms.
MODEL=/path/to/qwen4_exp_checkpoint

# Accuracy server, feature only:
.venv/bin/vllm serve "$MODEL" --port 8000 -tp 4 -ep \
  --enable-prefix-caching --no-enable-flashinfer-autotune \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

# Perf server: same command, replace --enable-prefix-caching with
# --numa-bind --no-enable-prefix-caching and set VLLM_USE_RUST_FRONTEND=1.
# Run baseline then feature, sequentially on the same node:
VLLM_USE_RUST_BENCH=1 .venv/bin/vllm bench serve \
  --backend openai-chat --base-url http://localhost:8000 \
  --model "$MODEL" --dataset-name speed-bench \
  --speed-bench-config throughput_8k --speed-bench-max-input-len 8192 \
  --speed-bench-output-len 1024 --ignore-eos --num-warmups 5 \
  --sweep-max-concurrency 1,4,16,64 --sweep-num-prompts-factor 10 \
  --percentile-metrics ttft,tpot,itl,e2el --metric-percentiles 50,90,99 \
  --save-result --result-dir results
```

</details>

### Static checks

Pre-commit on the changed files, manual `mypy-3.12`, and `git diff --check` passed. Rebased onto main `0c16eee3f1`; these static checks passed again. Full accuracy, A/B performance, and the official NVFP4 smoke at `0ed5afbd9c` were measured before this latest rebase.

**Review notes.** Duplicate checks found no projection-fusion PR; [#57097](https://github.com/aarnphm/aarnphm.github.io/issues/57097) fuses subsequent prepare operations, and [#59214](https://github.com/aarnphm/aarnphm.github.io/issues/59214) tunes existing SM100 shapes. AI assistance: OpenAI Codex ran the reported validation. Human review remains pending.

---

<details>
<summary> Pull Request Checklist </summary>

- [x] I used vLLM’s `/pr-checklist` skill. (Mandatory for agents, optional for humans).

- [x] AI assistance was used during the creation of this PR.

- [x] **Design Fit:** Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

- [x] **Testing and Validation:** Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

- [x] **Code Quality and Style:** Keeps code and comments clear and concise, and updates relevant documentation and examples.

- [x] **Pull Request Contents:** Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

</details>

---

## comments

**mergify** (2026-10-01T04:09:19Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @gau-nernst.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**gau-nernst** (2026-10-01T08:30:28Z):

@namgyu-youn @zigzagcai Since you added tuned LL-GEMM tables for SM100 and SM90, do you mind tuning them again for the merged GEMM in this PR? Send me the table, I will add you as co-author. If you don’t have time, I can merge first and you can add the tuned entries in a separate PR.

**namgyu-yun** (2026-10-01T08:38:40Z):

> @namgyu-youn @zigzagcai Since you added tuned LL-GEMM tables for SM100 and SM90, do you mind tuning them again for the merged GEMM in this PR? Send me the table, I will add you as co-author. If you don’t have time, I can merge first and you can add the tuned entries in a separate PR.

Sure I can help with SM100 (B200), let me share the table when it’s ready.

**zigzagcai** (2026-10-01T11:10:37Z):

> @namgyu-youn @zigzagcai Since you added tuned LL-GEMM tables for SM100 and SM90, do you mind tuning them again for the merged GEMM in this PR? Send me the table, I will add you as co-author. If you don’t have time, I can merge first and you can add the tuned entries in a separate PR.

@gau-nernst Sure. I can help to tune the table for SM90(H200) and share the table when it is ready.

**namgyu-yun** (2026-10-01T21:40:44Z):

@gau-nernst B200 table for the merged GEMM. The best configs match your GB300 entry; M>=8 stays on cuBLAS.



```python
    # QSA fused QKV/gate + replicated indexer Q/K, TP=4.
    (4224, 2560): {
        1: SkinnyGemmConfig(1, 128, 4, vector_width=4, static_k=2560),
        2: SkinnyGemmConfig(2, 128, 2, vector_width=4, static_k=2560),
        4: SkinnyGemmConfig(4, 64, 4, vector_width=4, static_k=2560),
    },
```

FlashInfer CUPTI + CUDA graphs. `separate` = main’s SM100 plans for (3584, 2560) + (640, 2560).



```csv
M,separate_cold_us,merged_cublas_cold_us,merged_ll_cold_us,ll_vs_cublas_cold,ll_vs_separate_cold,separate_hot_us,merged_cublas_hot_us,merged_ll_hot_us,ll_vs_cublas_hot,ll_vs_separate_hot,merged_ll_config
1,8.86,10.37,6.77,1.53,1.31,4.48,8.99,3.07,2.93,1.46,"(1,128,4,1,4,2560)"
2,9.14,8.74,6.69,1.31,1.37,5.46,7.49,4.22,1.77,1.29,"(2,128,2,1,4,2560)"
4,10.30,8.66,7.74,1.12,1.33,8.29,7.58,4.99,1.52,1.66,"(4,64,4,1,4,2560)"
8,13.07,8.83,8.80,1.00,1.49,10.72,7.68,7.68,1.00,1.40,"cuBLAS"
16,13.68,7.94,7.97,1.00,1.72,12.51,5.02,5.02,1.00,2.49,"cuBLAS"
64,13.36,8.10,8.06,1.00,1.66,10.08,5.28,5.28,1.00,1.91,"cuBLAS"
```

**gau-nernst** (2026-10-02T08:22:49Z):

@stecasta I will ask my agent to cover and validate MXFP8.

**mergify** (2026-10-02T17:28:23Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @gau-nernst.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**gau-nernst** (2026-10-05T03:48:45Z):

/ci run

**github-actions** (2026-10-05T03:49:04Z):

✅ Triggered [Buildkite CI #92828](https://buildkite.com/vllm/ci/builds/92828) for commit `11b9bad58f69`.

**vllm-agent** (2026-10-05T03:54:42Z):

<!-- ci-selector-shadow -->

### CI selector (shadow): 3 test steps (3 jobs) instead of 36 (51 jobs)

Shadow mode: this changes nothing about what CI runs. It shows what the evidence-based selector would pick for this PR, next to today’s rules. [How it works](https://github.com/vllm-project/ci-infra/tree/main/buildkite/ci_selector).

**Feedback welcome:** reply here if it would skip a step this change needs, or runs something unrelated.

| steps (jobs)           | Today’s rules | Selector | Would skip | Would add |
| ---------------------- | ------------- | -------- | ---------- | --------- |
| NVIDIA, CPU and others | 36 (51)       | 3 (3)    | 33 (48)    | 0 (0)     |
| AMD mirrors            | 33 (43)       | 1 (1)    | 32 (42)    | 0 (0)     |

<details><summary>Selector would run (3)</summary>

- `kernels-root-misc-test-b200`
- `qwen4-exp-unit-tests`
- `qwen4-exp-unit-tests-cpu`

</details>

<details><summary>Would skip (today's rules run them) (33)</summary>

- `ascend-npu-test`
- `basic-correctness` ×2
- `basic-correctness-cpu-offload`
- `basic-correctness-cumem`
- `basic-correctness-prefetch-offload`
- `basic-correctness-sleep-mode`
- `basic-models-test-other-cpu`
- `basic-models-tests-initialization`
- `basic-models-tests-other`
- `benchmarks-cli-test`
- `cpu-language-generation-and-pooling-model-tests` ×3
- `distributed-model-tests-2-gpus` ×3
- `entrypoints-integration-api-server` ×4
- `entrypoints-integration-api-server-generate`
- `entrypoints-integration-api-server-openai-chat_completion`
- `entrypoints-integration-api-server-openai-completion`
- `entrypoints-integration-llm`
- `entrypoints-integration-multimodal`
- `entrypoints-integration-pooling`
- `entrypoints-integration-responses-api`
- `entrypoints-integration-speech_to_text`
- `kernels-fla-ops-test-b200`
- `kernels-mhc-test-b200`
- `language-models-tests-granite-l4-compatibility`
- `language-models-tests-hybrid` ×2
- `language-models-tests-standard`
- `multi-modal-models-standard-1-qwen2`
- `multi-modal-models-standard-2-qwen3-gemma`
- `multi-modal-models-standard-3-llava-qwen2-vl`
- `multi-modal-models-standard-4-other-whisper`
- `multi-modal-processor` ×4
- `multi-modal-processor-cpu` ×4
- `v1-spec-decode`

</details>

<details><summary>Would add (today's rules do not run them) (0)</summary>

none

</details>

<details><summary>AMD mirrors: would skip (32)</summary>

- `basic-correctness` ×2
- `basic-correctness-cpu-offload`
- `basic-correctness-cumem`
- `basic-correctness-prefetch-offload`
- `basic-correctness-sleep-mode`
- `basic-models-tests-initialization`
- `basic-models-tests-other`
- `benchmarks-cli-test`
- `distributed-model-tests-2-gpus` ×3
- `entrypoints-integration-api-server` ×4
- `entrypoints-integration-api-server-generate`
- `entrypoints-integration-api-server-openai-chat_completion`
- `entrypoints-integration-api-server-openai-completion`
- `entrypoints-integration-llm`
- `entrypoints-integration-multimodal`
- `entrypoints-integration-pooling`
- `entrypoints-integration-responses-api`
- `entrypoints-integration-speech_to_text`
- `kernels-fla-ops-test-b200`
- `kernels-mhc-test-b200`
- `language-models-tests-granite-l4-compatibility`
- `language-models-tests-hybrid` ×2
- `language-models-tests-standard`
- `multi-modal-models-standard-1-qwen2`
- `multi-modal-models-standard-2-qwen3-gemma`
- `multi-modal-models-standard-3-llava-qwen2-vl`
- `multi-modal-models-standard-4-other-whisper`
- `multi-modal-processor` ×4
- `platform-tests`
- `pytorch-compilation-passes-unit-tests`
- `qwen4-exp-unit-tests`
- `v1-spec-decode`

</details>

<details><summary>AMD mirrors: would add (0)</summary>

none

</details>

<sub>6 changed files · base `92044241a0` · head `9c6bc89258` · Python record: build 92817 at `7867d6c52d` · kernel record: table [`7867d6c`](https://github.com/aarnphm/aarnphm.github.io/commit/7867d6c52d) (build 92817), map [`7867d6c`](https://github.com/aarnphm/aarnphm.github.io/commit/7867d6c52d) · not counted: 10 build steps, 5 A100 steps the generator no longer emits, 2 optional steps the selector would also run</sub>

**gau-nernst** (2026-10-05T05:54:48Z):

/ci run

**github-actions** (2026-10-05T05:55:07Z):

✅ Triggered [Buildkite CI #92837](https://buildkite.com/vllm/ci/builds/92837) for commit `9c6bc892582b`.

**zigzagcai** (2026-10-05T07:31:38Z):

@gau-nernst Sorry for the late response. Just come back from vacation and at my desktop.

Here is my LL-GEMM tune table for H200(SM90).



```
H200/SM90 table for the merged GEMM. M>=8 stays on cuBLAS.

# QSA fused QKV/gate + replicated indexer Q/K, TP=4.
(4224, 2560): {
    1: SkinnyGemmConfig(1, 128, 2, k_unroll=4,
                        vector_width=4, static_k=2560),
    2: SkinnyGemmConfig(2, 64, 4, k_unroll=2,
                        vector_width=4, static_k=2560),
    4: SkinnyGemmConfig(4, 64, 4, k_unroll=2,
                        vector_width=4, static_k=2560),
},

FlashInfer CUPTI + CUDA graphs. `separate` = main's SM90 plans for (3584, 2560) + (640, 2560).

CSV columns:

M,separate_cold_us,merged_cublas_cold_us,merged_ll_cold_us,ll_vs_cublas_cold,ll_vs_separate_cold,separate_hot_us,merged_cublas_hot_us,merged_ll_hot_us,ll_vs_cublas_hot,ll_vs_separate_hot,merged_ll_config
1,10.78,9.63,8.00,1.20,1.35,6.21,6.56,4.09,1.60,1.52,"(1,128,2,4,4,2560)"
2,11.20,9.63,8.48,1.14,1.32,7.81,6.27,4.54,1.38,1.72,"(2,64,4,2,4,2560)"
4,13.06,9.63,8.96,1.08,1.46,10.11,6.11,5.92,1.03,1.71,"(4,64,4,2,4,2560)"
8,13.54,9.63,9.63,1.00,1.41,9.66,6.14,6.14,1.00,1.57,"cuBLAS"
16,15.65,9.86,9.87,1.00,1.59,11.65,5.54,5.54,1.00,2.10,"cuBLAS"
64,16.16,9.79,9.82,1.00,1.64,12.51,6.43,6.43,1.00,1.94,"cuBLAS"
```

I see H200 tuning table already been commited in <https://github.com/vllm-project/vllm/pull/59533/changes/639add00836feba35f9627ed749a7cebde8d9776>

**gau-nernst** (2026-10-05T07:37:33Z):

@zigzagcai no worries, I found a H200 node and did the tuning there. If you find my tuned results bad, feel free to open a new PR.

**zigzagcai** (2026-10-09T05:12:52Z):

> @zigzagcai no worries, I found a H200 node and did the tuning there. If you find my tuned results bad, feel free to open a new PR.

@gau-nernst Thanks! I cross-checked the H200(SM90) tuning table and found that the configuration in your commit is already optimal for M=1 and M=2, though there is a slightly better configuration for the M=4 scenario. So I open a new PR dedicated for the M=4 optimal case, [vllm-project/vllm#60776](https://github.com/vllm-project/vllm/pull/60776)

