---
title: '54261'
pageLayout: default
slug: thoughts/inference/vllm-prs/54261
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/54261.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#54261](https://github.com/aarnphm/aarnphm.github.io/issues/54261) \[Kimi-K3\]\[Perf\] Make native CUDA AttnRes the SM100 default

[vllm-project/vllm#54261](https://github.com/vllm-project/vllm/pull/54261) · zyongye · MERGED · created 2026-08-28T19:46:08Z · merged 2026-08-31T23:18:12Z

## Purpose

Make the native CUDA Kimi-K3 AttnRes implementation the default for dense
`hidden_size=7168` inputs on SM100.

This PR:

- extends the existing register/TMEM kernel to cover block counts 0 through 8,
  optional prefix addition, optional output normalization, and block-bank
  writes;
- uses the existing source-chunk schedule for all supported block counts,
  while preserving a compile-time-folded common path;
- makes output normalization a compile-time specialization so the disabled
  path elides its shared-memory buffer, barrier, and arithmetic;
- delays the PDL wait until the first producer-dependent prefix or delta TMA
  for every native shape and triggers dependents after the producer TMAs;
- retains Triton as the fallback for unsupported devices and non-dense row
  layouts; and
- adds a reproducible CUDA-versus-Triton kernel benchmark.

Duplicate-work check: the author of [#53524](https://github.com/aarnphm/aarnphm.github.io/issues/53524) coordinated moving its AttnRes PDL
handoff into this integrated change; that PR’s remaining distinct scope is its
C=1 MoE/router work. [#50760](https://github.com/aarnphm/aarnphm.github.io/issues/50760) changes warmup profiles only. Neither otherwise
extends the native kernel to all Kimi-K3 AttnRes call variants.

## Performance

Measured on GPU 7, an NVIDIA B300 SXM6 AC, with PyTorch 2.13.0 and CUDA 13.0.
Times are median microseconds; lower is better.

| Tokens | Blocks | Triton |   CUDA | Speedup |
| -----: | -----: | -----: | -----: | ------: |
|      1 |      0 |   8.19 |   8.19 |   1.00x |
|      1 |      1 |  10.24 |   8.19 |   1.25x |
|      1 |      4 |  12.29 |   8.22 |   1.49x |
|      1 |      8 |  14.34 |  10.24 |   1.40x |
|   4096 |      8 | 204.77 | 122.88 |   1.67x |
|   8192 |      8 | 395.30 | 229.44 |   1.72x |

For the measured `M <= 16` cases with one or more block rows, CUDA was 1.20x
to 1.49x faster than Triton. The zero-block path tied Triton at 8.19 us.

In a captured single-token CUDA graph, profiler device rows for the CUDA
kernel were 3.67, 4.44, and 5.48 us for 1, 4, and 8 block rows. Full graph
replay medians were 8.03, 10.05, and 10.05 us, respectively.

The no-output-normalization specialization was neutral for the measured
`M <= 16` cases and up to 4.6% faster for larger measured shapes. For example,
the 8-block, M=1024 case improved from 40.86 to 38.98 us.

## Test Plan



```bash
cmake --build cmake-build-release --target _C_stable_libtorch
cmake --install cmake-build-release --component _C_stable_libtorch

CUDA_VISIBLE_DEVICES=7 .venv/bin/python -m pytest \
  tests/models/kimi_k3/test_attn_res.py \
  tests/models/kimi_k3/test_aux_attn_res_stream.py \
  tests/models/kimi_k3/test_amd_attn_res.py -q

CUDA_VISIBLE_DEVICES=7 .venv/bin/python \
  benchmarks/kernels/benchmark_kimi_k3_attn_res.py \
  --num-tokens 1 16 1024 4096 8192 \
  --num-blocks 0 1 4 8 --warmup-ms 50 --repeat-ms 200

.venv/bin/pre-commit run --files \
  benchmarks/kernels/benchmark_kimi_k3_attn_res.py \
  csrc/libtorch_stable/kimi_k3/attn_res_kernel.cu \
  csrc/libtorch_stable/ops.h \
  csrc/libtorch_stable/torch_bindings.cpp \
  tests/models/kimi_k3/test_attn_res.py \
  vllm/_custom_ops.py \
  vllm/models/kimi_k3/nvidia/ops/attn_res.py
```

## Test Result

- Incremental CUDA build: passed.
- Focused Kimi-K3 suite: 34 passed, 9 platform skips.
- Benchmark reference checks for 0, 1, 4, and 8 blocks: passed.
- All pre-commit hooks on changed files: passed.
- Model evaluation: run on the full Kimi-K3 checkpoint across five serving
  configurations; see Model Evaluation below. Kernel outputs and in-place
  mutations are additionally checked against the PyTorch reference across all
  block counts and fused variants.

## Model Evaluation

Full `moonshotai/Kimi-K3` (MXFP4) on GB300 (SM100), vLLM
`0.17.2rc1.dev5039+g73b83949f`, across five serving configurations. GSM8K is
5-shot via `/v1/completions` (temperature 0.6, top_p 0.95); OCRBench is
multimodal chat with thinking effort high (temperature 1.0, top_p 1.0).
Reported scores are GSM8K strict-match exact match and OCRBench Final Score
Norm.

| Config                   | Topology                |           GSM8K |        OCRBench | Request errors |
| ------------------------ | ----------------------- | --------------: | --------------: | -------------: |
| TP8                      | TP8                     | 95.53% (n=1319) | 89.3 (893/1000) |              0 |
| TEP8                     | TP8 + EP                | 97.27% (n=1319) | 89.3 (893/1000) |              0 |
| DEP16                    | TP1 x DP16 x EP16       | 95.98% (n=1319) | 89.2 (892/1000) |              0 |
| TP8, `--max-num-seqs 16` | TP8, eval concurrency 8 | 96.36% (n=1319) |             n/a |              0 |
| TP8, serial              | TP8, eval concurrency 1 |  99.60% (n=250) |             n/a |              0 |

No accuracy regression. GSM8K spans 95.5% to 97.3% across topologies, which is
within run-to-run variance for temperature-0.6 sampling (roughly +/-0.5 point
standard error at n=1319). OCRBench agrees to within 0.1 point across the three
multimodal configurations.

The two low-concurrency rows are the most direct check on this change: they
hold decode batches at 1 to 16 tokens, the small-batch regime this PR moves
from Triton onto the native kernel. The serial row’s 99.60% reflects its
250-sample subset rather than a topology difference — scoring the TP8 run’s
own outputs on those identical 250 items gives 98.80%, a two-item difference.

AI assistance disclosure: AI assistance was used for implementation,
benchmarking, testing, and PR preparation. This PR is left as a draft until the
human submitter reviews every changed line and confirms they can defend the
change end-to-end.

---

<details>
<summary> Essential Elements of an Effective PR Description Checklist </summary>

- [x] The purpose of the PR is described.
- [x] Test commands and results are provided.
- [x] Correctness and performance results are provided.
- [x] No documentation update is needed for this kernel-only change.

</details>

---

## comments

**zyongye** (2026-08-30T07:24:10Z):

/ci run

**github-actions** (2026-08-30T07:24:27Z):

✅ Triggered [Buildkite CI #86201](https://buildkite.com/vllm/ci/builds/86201) for commit `7caf67d54655`.

**zyongye** (2026-08-31T06:28:59Z):

/ci run

**github-actions** (2026-08-31T06:29:17Z):

✅ Triggered [Buildkite CI #86299](https://buildkite.com/vllm/ci/builds/86299) for commit `91ddd3647f4d`.

**zyongye** (2026-08-31T21:12:24Z):

/ci run

**github-actions** (2026-08-31T21:12:43Z):

✅ Triggered [Buildkite CI #86425](https://buildkite.com/vllm/ci/builds/86425) for commit `2d2c75d2dfc7`.

