---
title: '59069'
pageLayout: default
slug: thoughts/inference/vllm-prs/59069
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/59069.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#59069](https://github.com/aarnphm/aarnphm.github.io/issues/59069) \[ROCm\]\[Kimi-K3\] Fuse AttnRes output with per-token FP8 input quantization

[vllm-project/vllm#59069](https://github.com/vllm-project/vllm/pull/59069) · whx-sjtu · MERGED · created 2026-09-28T16:23:18Z · merged 2026-10-08T12:01:48Z

## Summary

Eligible Kimi-K3 AMD AttnRes outputs currently materialize BF16 activations that the next linear immediately quantizes. Fuse that per-token FP8 conversion into AttnRes and pass native `QuantizedActivation` to supported consumers.



```diff
 AttnRes + output RMSNorm
-  store BF16 → per-token FP8 quant → input GEMM
+  store FP8 + scale → input GEMM
```

Enable the path only for `kFp8DynamicTokenSym` and only when all consumers accept the same key. Mixed MoE/router consumers, unsupported linear kernels, and batch-invariant mode retain the original path. Preserve BF16 rounding before quantization and the prefix/block write contract.

This is independent of PD connector work and targets main. Duplicate checks covered [#50682](https://github.com/aarnphm/aarnphm.github.io/issues/50682), [#52968](https://github.com/aarnphm/aarnphm.github.io/issues/52968), and [#53154](https://github.com/aarnphm/aarnphm.github.io/issues/53154): [#52968](https://github.com/aarnphm/aarnphm.github.io/issues/52968)’s current diff covers convolution and sigmoid/multiply; [#53154](https://github.com/aarnphm/aarnphm.github.io/issues/53154) covers MXFP4 output fusions. This PR supplies AttnRes per-token FP8 inputs. The current rowwise Torch FP8 KDA kernel does not expose the required input capability, so this PR does not claim all KDA inputs are fused.

## Evidence

### CI follow-up (2026-10-08)

[Buildkite #93497](https://buildkite.com/vllm/ci/builds/93497) exposed an implicit compiled reference in the FP8 output test when AITER is disabled. Commit `42f0ad593` sets `compile_native=False` for the `QuantFP8` reference, keeping its rounding independent of Inductor fusion. Production code and every numerical tolerance are unchanged. The AttnRes and MLA/KDA direct-return suites pass with AITER disabled (**23 passed**) and enabled (**23 passed**); changed-file pre-commit and Python 3.12 mypy pass. [Buildkite #93546](https://buildkite.com/vllm/ci/builds/93546) retries the failed K3 step on that commit.

- **Before:** AttnRes and per-token quantization launch separately.
  **After:** A single producer emits activation and scale. Existing + new AttnRes cases passed on all eight MI355X GPUs (16 tests/rank in the final run, including the three capability cases), with exact prefix/block writes, scale rtol 1e-6, and rounding differences limited to an adjacent FP8 code in <1e-4 of values. Three direct-return tests also passed.
- In the paired rank-0 prefill trace, the corresponding standalone AITER per-token quant kernel count falls from 4009 to 3078 (931 calls removed), while AttnRes executes the same 3553 calls.
- `pre-commit run` and `pre-commit run mypy-3.12 --hook-stage manual`: passed.
- Eight-rank `benchmarks/kernels/benchmark_kimi_k3_amd_fp8_fusions.py` compares AttnRes + quant + one MLA input GEMM (N2112/K7168), using nine rotated CUDA-graph samples per rank. Median of per-rank medians, four residual blocks:

| Tokens | Before (µs) | After (µs) |
| -----: | ----------: | ---------: |
|     16 |      17.110 |     16.597 |
|    256 |      31.701 |     30.168 |
|   4096 |     183.573 |    173.933 |
|  16384 |     682.589 |    657.554 |

This is a producer/projection segment benchmark. One-token results are essentially unchanged; it does not establish a PD throughput gain.

Paired full-model integration validation used the same eight MI355X GPUs, TP8/DCP8, DSpark3 with natural/standard rejection, FULL_AND_PIECEWISE, 80 sequences / 16384 batched tokens / [`1048576`](https://github.com/aarnphm/aarnphm.github.io/commit/1048576) context. The candidate adds both this PR and the companion DCP prefill fusion [#59070](https://github.com/aarnphm/aarnphm.github.io/issues/59070) to an exact saved diagnostic baseline; the following model results are combined, not isolated attribution to this PR.

- Three unprofiled repetitions of the same eight-case prefix chain: all 24 first-token outputs, cache-hit counts and computed-token counts match. Median TTFT: cold 32K 1701.03 → 1673.86 ms; exact 32K repeat 83.37 → 77.13 ms; the two other repeats 69.97 → 64.42 and 70.79 → 64.56 ms. The 1024-token append is essentially unchanged (290.44 → 291.09 ms).
- Natural-acceptance C16 decode diagnostic: TPOT 9.867 → 9.840 ms. This is a local single-instance test, not the formal synthetic-AL3 PD C16 benchmark.
- GSM8K, first 128 test examples, 5-shot, seed `0,1234,1234,1234`, concurrency 16, greedy generation, max 8192 output tokens: **127/128 (99.21875%) before and after**, for both extractors. The missed example changes (baseline doc 125; candidate doc 84). Aggregate accuracy is unchanged, but this does not establish per-example equivalence or a full-dataset accuracy result.
- Eight 33K+ context checks (cold and cached): every final arithmetic question is answered correctly. Full continuations match in 7/8 cases; the remaining continuation differs after the correct answer. Outputs/logprobs are not bit-exact.

Evaluation command:



```bash
python -m lm_eval run --model local-completions \
  --model_args 'model=Kimi-K3,base_url=http://127.0.0.1:2584/v1/completions,tokenizer_backend=None,tokenized_requests=False,num_concurrent=16,max_retries=1,timeout=1800,max_length=1048576,max_gen_toks=8192' \
  --tasks gsm8k --num_fewshot 5 --batch_size 1 --limit 128 \
  --gen_kwargs 'max_gen_toks=8192,temperature=0.0,do_sample=False' \
  --seed 0,1234,1234,1234 --log_samples --output_path results
```

Full-dataset accuracy and isolated per-PR model measurements remain review work before merging.

Kernel test/benchmark commands (GPU tests were assigned one pytest process per rank across all eight devices):



```bash
python -m pytest tests/models/kimi_k3/test_amd_attn_res.py -q
python -m torch.distributed.run --nproc-per-node=8 \
  benchmarks/kernels/benchmark_kimi_k3_amd_fp8_fusions.py --output results
```

## Merge Danger

**Door:** Two-way

No checkpoint or cache format change. Reverting restores the original computation path.

**Blast Radius:** ROCm

AI assistance (OpenAI Codex) was used for implementation, analysis, and validation.

---

## comments

**mergify** (2026-09-30T23:19:40Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @whx-sjtu.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**tjtanaa** (2026-10-01T04:05:33Z):

overall LGTM. Can you rebase @whx-sjtu resolve merge conflict and check if your changes still provide perf gain?

**whx-sjtu** (2026-10-01T07:12:00Z):

> overall LGTM. Can you rebase @whx-sjtu resolve merge conflict and check if your changes still provide perf gain?

@tjtanaa rebased, and the perf gain remains. Can we start ci run now?

**tjtanaa** (2026-10-01T14:20:04Z):

/ci run

**github-actions** (2026-10-01T14:20:15Z):

✅ @whx-sjtu, CI is now available for this PR.

- `/ci run` starts upstream CI; `/amd-ci run` starts AMD CI only.
- Your branch must contain every commit currently on its upstream target branch. Merge or rebase onto the latest target branch, then rerun the command. Append `--allow-stale` to a run command to test an outdated branch at your own risk.
- `/ci retry` retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
- `/amd-ci retry` retries failed jobs in AMD CI for the current PR head. Use `/amd-ci run` when the current head has no AMD CI build.
- `/ci cancel` cancels scheduled or running CI builds for this PR branch; `/amd-ci cancel` does the same for AMD CI only.

<!-- vllm-ci-authorized -->

**github-actions** (2026-10-01T14:20:25Z):

❌ This PR is 21 commits behind upstream `main`. Your branch must contain every commit currently on upstream `main`. No new CI build was started. Merge or rebase onto the latest `main`, then rerun `/ci run`. To test this branch at your own risk, use `/ci run --allow-stale`.

<!-- vllm-ci-command:5933387231 -->

**whx-sjtu** (2026-10-03T14:14:13Z):

/ci run

**github-actions** (2026-10-03T14:14:32Z):

❌ This PR is 127 commits behind upstream `main`. Your branch must contain every commit currently on upstream `main`. No new CI build was started. Merge or rebase onto the latest `main`, then rerun `/ci run`. To test this branch at your own risk, use `/ci run --allow-stale`.

<!-- vllm-ci-command:5969986529 -->

**whx-sjtu** (2026-10-03T14:16:16Z):

/ci run

**github-actions** (2026-10-03T14:16:35Z):

✅ Triggered [Buildkite CI #92758](https://buildkite.com/vllm/ci/builds/92758) for commit `bc28cc4bf185`.

**vllm-agent** (2026-10-03T14:28:51Z):

<!-- ci-selector-shadow -->

### CI selector (shadow): 2 test steps (2 jobs) instead of 36 (52 jobs)

Shadow mode: this changes nothing about what CI runs. It shows what the evidence-based selector would pick for this PR, next to today’s rules. [How it works](https://github.com/vllm-project/ci-infra/tree/main/buildkite/ci_selector).

**Feedback welcome:** reply here if it would skip a step this change needs, or runs something unrelated.

| steps (jobs)           | Today’s rules | Selector | Would skip | Would add |
| ---------------------- | ------------- | -------- | ---------- | --------- |
| NVIDIA, CPU and others | 36 (52)       | 2 (2)    | 35 (51)    | 1 (1)     |
| AMD mirrors            | 33 (43)       | 11 (33)  | 28 (35)    | 6 (25)    |

<details><summary>Selector would run (2)</summary>

- `cpu-spec-decode-tests`
- `kimi-k3-unit-tests-b200`

</details>

<details><summary>Would skip (today's rules run them) (35)</summary>

- `ascend-npu-test`
- `basic-correctness` ×2
- `basic-correctness-cpu-offload`
- `basic-correctness-cumem`
- `basic-correctness-prefetch-offload`
- `basic-correctness-sleep-mode`
- `basic-models-test-other-cpu`
- `basic-models-tests-initialization`
- `basic-models-tests-other`
- `benchmarks-cli-test`
- `cpu-language-generation-and-pooling-model-tests` ×3
- `distributed-model-tests-2-gpus` ×3
- `entrypoints-integration-api-server` ×4
- `entrypoints-integration-api-server-generate`
- `entrypoints-integration-api-server-openai-chat_completion`
- `entrypoints-integration-api-server-openai-completion`
- `entrypoints-integration-llm`
- `entrypoints-integration-multimodal`
- `entrypoints-integration-pooling`
- `entrypoints-integration-responses-api`
- `entrypoints-integration-speech_to_text`
- `kernels-fla-ops-test-b200`
- `kernels-mhc-test-b200`
- `kernels-root-misc-test-b200`
- `kimi-k3-prefix-cache-4xb200` ×2
- `language-models-tests-granite-l4-compatibility`
- `language-models-tests-hybrid` ×2
- `language-models-tests-standard`
- `multi-modal-models-standard-1-qwen2`
- `multi-modal-models-standard-2-qwen3-gemma`
- `multi-modal-models-standard-3-llava-qwen2-vl`
- `multi-modal-models-standard-4-other-whisper`
- `multi-modal-processor` ×4
- `multi-modal-processor-cpu` ×4
- `rl-entrypoints-tests`

</details>

<details><summary>Would add (today's rules do not run them) (1)</summary>

- `cpu-spec-decode-tests` (code map)

</details>

<details><summary>AMD mirrors: would skip (28)</summary>

- `basic-correctness` ×2
- `basic-correctness-cpu-offload`
- `basic-correctness-cumem`
- `basic-correctness-prefetch-offload`
- `basic-correctness-sleep-mode`
- `basic-models-tests-initialization`
- `benchmarks-cli-test`
- `distributed-model-tests-2-gpus` ×3
- `entrypoints-integration-api-server` ×4
- `entrypoints-integration-api-server-generate`
- `entrypoints-integration-api-server-openai-chat_completion`
- `entrypoints-integration-api-server-openai-completion`
- `entrypoints-integration-llm`
- `entrypoints-integration-multimodal`
- `entrypoints-integration-pooling`
- `entrypoints-integration-responses-api`
- `entrypoints-integration-speech_to_text`
- `kernels-fla-ops-test-b200`
- `kernels-mhc-test-b200`
- `language-models-tests-granite-l4-compatibility`
- `language-models-tests-hybrid` ×2
- `language-models-tests-standard`
- `multi-modal-models-standard-1-qwen2`
- `multi-modal-models-standard-2-qwen3-gemma`
- `multi-modal-models-standard-3-llava-qwen2-vl`
- `multi-modal-models-standard-4-other-whisper`
- `platform-tests`
- `pytorch-compilation-passes-unit-tests`

</details>

<details><summary>AMD mirrors: would add (6)</summary>

- `basic-models-tests-extra-initialization` ×14 (code map)
- `batch-invariance-h100` (code map)
- `kernels-attention-test` ×7 (code map)
- `pytorch-nightly-dependency-override-check` (code map)
- `ray-dependency-compatibility-check` (code map)
- `torch-stable-abi-audit` (code map)

</details>

<sub>7 changed files · base `87d9996abe` · head `42f0ad5930` · Python record: build 93039 at `1e5d0ea888` · kernel record: table [`a982c81`](https://github.com/aarnphm/aarnphm.github.io/commit/a982c81e3e) (build 93415), map [`a982c81`](https://github.com/aarnphm/aarnphm.github.io/commit/a982c81e3e) · not counted: 10 build steps, 5 A100 steps the generator no longer emits, 9 optional steps the selector would also run</sub>

**Fangzhou-Ai** (2026-10-08T03:05:42Z):

/ci run

**github-actions** (2026-10-08T03:06:01Z):

✅ Triggered [Buildkite CI #93497](https://buildkite.com/vllm/ci/builds/93497) for commit `ed4ce96e154e`.

**whx-sjtu** (2026-10-08T07:56:32Z):

/ci retry

**github-actions** (2026-10-08T07:56:55Z):

✅ Triggered [Buildkite CI #93546](https://buildkite.com/vllm/ci/builds/93546) for commit `42f0ad59301b`, running 1 failed step(s) from [Buildkite CI #93497](https://buildkite.com/vllm/ci/builds/93497).

