---
title: '60717'
pageLayout: default
slug: thoughts/inference/vllm-prs/60717
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/60717.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#60717](https://github.com/aarnphm/aarnphm.github.io/issues/60717) \[ROCm\]\[Kimi-K3\] Use vLLM’s FP8 max in the fused AttnRes quantization

[vllm-project/vllm#60717](https://github.com/vllm-project/vllm/pull/60717) · Rohan138 · MERGED · created 2026-10-08T21:06:55Z · merged 2026-10-09T04:45:13Z

## Overview

Fixes `test_amd_attn_res_fp8_preserves_prefix_and_quantized_output` on MI300 (fnuz FP8), which has failed since [#59069](https://github.com/aarnphm/aarnphm.github.io/issues/59069) landed.

## Claims

- The fused AttnRes per-token FP8 quantization now uses `get_fp8_min_max()` (224 on fnuz) instead of `torch.finfo(dtype).max` (240), matching vLLM’s `QuantFP8` and the unfused path before [#59069](https://github.com/aarnphm/aarnphm.github.io/issues/59069). MI355 (OCP `e4m3fn`, 448 either way) is unchanged.
- The test’s FP8 code-distance check orders codes by value, so a midpoint step across zero (`0x00` vs `0x81`) counts as one step rather than 129.

## Validation

MI300X (gfx942), AMD CI test image for [#58761](https://github.com/aarnphm/aarnphm.github.io/issues/58761) (`rocm/vllm-ci:build-01a11cfd-…`, ROCm 10). Both changed files are identical on main.

| `pytest models/kimi_k3/test_amd_attn_res.py` | Result                                                   |
| -------------------------------------------- | -------------------------------------------------------- |
| main                                         | 3 failed, 13 passed (same 3 cases as CI)                 |
| kernel fix only                              | 1 failed (`[320-8-False-False]`, the zero-crossing case) |
| kernel + test fix                            | 16 passed                                                |

The MI300 Kimi K3 CI job’s full test set (`test_amd_attn_res.py test_amd_kda_decode.py test_kda.py test_kda_metadata.py`) passes: 139 passed, 30 skipped. I did not run an end-to-end Kimi-K3 eval (no checkpoint or hardware here). On MI300 this restores the 224 scale convention that `QuantFP8` used before the fusion.

## Details

On fnuz, every scale the fused kernel produced was 240/224 of `QuantFP8`’s (relative diff exactly 1/15). The remaining failure was a value at the rounding midpoint near zero (−0.000484 vs midpoint −0.000488): the kernel quantizes its fp32 result, the reference quantizes the bf16-rounded one, so they land on adjacent codes on opposite sides of zero.

Not a duplicate: no open PR changes this. [#58861](https://github.com/aarnphm/aarnphm.github.io/issues/58861) touches the same `QUANT_MAX` line as context and will need a trivial rebase.

---

<details>
<summary> Pull Request Checklist </summary>

- [x] I used vLLM’s `/pr-checklist` skill. (Mandatory for agents, optional for humans).

- [x] AI assistance was used during the creation of this PR.

- [x] **Design Fit:** Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

- [x] **Testing and Validation:** Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

- [x] **Code Quality and Style:** Keeps code and comments clear and concise, and updates relevant documentation and examples.

- [x] **Pull Request Contents:** Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

</details>

---

## comments

**AndreasKaratzas** (2026-10-08T21:24:06Z):

/ci run

**github-actions** (2026-10-08T21:24:26Z):

✅ Triggered [Buildkite CI #93718](https://buildkite.com/vllm/ci/builds/93718) for commit `f72d5ee12623`.

**AndreasKaratzas** (2026-10-09T02:02:24Z):

For some reason, this one : <https://buildkite.com/vllm/ci/builds/93718/list?jid=01a11e39-3634-42b2-ab90-ca6202e14f07&tab=output> timed out **twice** .. is there any different code path that is taken here or something? We are also going to investigate whether we just need to increment our timeouts, but check it out, maybe it’s a regression.

**AndreasKaratzas** (2026-10-09T02:47:48Z):

Okay, it might not be an issue with this PR. We are still working things out in DPX.

