---
title: '50761'
pageLayout: default
slug: thoughts/inference/vllm-prs/50761
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/50761.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#50761](https://github.com/aarnphm/aarnphm.github.io/issues/50761) \[ROCm\]\[Bugfix\]\[Kimi-K3\] Preserve MoE correction bias in FP32

[vllm-project/vllm#50761](https://github.com/vllm-project/vllm/pull/50761) · Fangzhou-Ai · MERGED · created 2026-08-02T18:30:41Z · merged 2026-08-02T20:30:55Z

## Purpose

Kimi-K3 checkpoints store `e_score_correction_bias` in FP32 and the AMD router already emits FP32 logits. The AMD model currently constructs the correction-bias parameter under the ambient BF16 model dtype, so the standard weight loader copies the FP32 checkpoint into BF16 storage. A later AITER runtime upcast cannot restore the discarded low bits.

Construct the small correction-bias parameter in FP32 before weight loading. This preserves checkpoint precision, matches the router-logit contract, and avoids repeatedly converting the bias before grouped top-k. The change mirrors Kimi-K3’s NVIDIA implementation and the merged DeepSeek fix in [#23640](https://github.com/aarnphm/aarnphm.github.io/issues/23640).

Related tracking issue: [#50682](https://github.com/aarnphm/aarnphm.github.io/issues/50682).

### Duplicate/overlap check

No exact open PR was found in the required searches.

- Draft [#50319](https://github.com/aarnphm/aarnphm.github.io/issues/50319) includes a post-load FP32 cast only in its opt-in gfx942 int4 setup. Because that cast occurs after the checkpoint was copied into BF16 storage, it cannot recover the lost checkpoint bits; it also does not cover the default gfx950 path. This PR allocates FP32 before loading and applies to all AMD Kimi-K3 paths.
- [#50268](https://github.com/aarnphm/aarnphm.github.io/issues/50268) changes the ROCm router GEMM’s FP32 output path; it does not change correction-bias storage.

## Test Plan

No unit test is added for this dtype declaration. The source change is four added lines/one removed line, matches the established NVIDIA declaration, and has the merged one-line DeepSeek precedent above. The output-affecting concern is instead validated with the full model evaluation.



```bash
pre-commit run \
  --files vllm/models/kimi_k3/amd/linear.py

lm_eval run \
  --model local-completions \
  --model_args 'model=moonshotai/Kimi-K3,base_url=http://127.0.0.1:8000/v1/completions,num_concurrent=64,max_retries=3,tokenized_requests=False,trust_remote_code=True,seed=42,max_gen_toks=256' \
  --tasks gsm8k --num_fewshot 5 --seed 42 --log_samples
```

The full GSM8K arm used TP8 on 8× MI355X, lm-eval 0.4.12, all 1,319 questions, five shots, raw completions, greedy decoding, a 256-token generation cap, and identical prompt/target hashes. Base and candidate each ran on a fresh server after the same C24 conditioning sequence.

## Test Result

Changed-file pre-commit: all hooks passed, including Ruff, formatting, mypy, SPDX, forbidden-import, and configuration checks. `git diff --check` passed.

Checkpoint metadata inspection confirmed the tested correction-bias tensor is `F32 [896]`.

### Constructor dtype verification

The original allocation does not intrinsically request BF16; it inherits PyTorch’s current default dtype. vLLM constructs and loads the model inside `set_default_torch_dtype(model_config.dtype)`, which is BF16 for the tested Kimi-K3 deployment. `GateLinear(..., out_dtype=torch.float32)` controls the router output only and does not affect the independent correction-bias allocation. The standard weight loader then uses `param.data.copy_(loaded_weight)`, which converts the FP32 checkpoint tensor to the destination parameter dtype.

A constructor-level runtime probe executed the actual AMD `KimiMoE.__init__` under the BF16 model context, substituting only lightweight gate and MoE factory stubs to avoid allocating the full model:

| Probe                                        | Original allocation | Explicit FP32 allocation |
| -------------------------------------------- | ------------------: | -----------------------: |
| Default dtype during initialization          |    `torch.bfloat16` |         `torch.bfloat16` |
| Router output contract                       |     `torch.float32` |          `torch.float32` |
| Correction bias after initialization         |    `torch.bfloat16` |          `torch.float32` |
| Correction bias after loading an FP32 tensor |    `torch.bfloat16` |          `torch.float32` |
| Exact equality with the FP32 source          |             `False` |                   `True` |

For representative non-BF16-exact values, the original path stored `[0.12353516, -0.23437500, 0.34570312, -0.45703125]` instead of `[0.12345679, -0.23456790, 0.34567890, -0.45678911]`. The explicit dtype preserves the source values exactly. This was a diagnostic runtime probe; no unit test is added to this one-line dtype fix.

| GSM8K filter     |     Base `060185079` | FP32 bias `407e5249d` |           Delta |
| ---------------- | -------------------: | --------------------: | --------------: |
| Strict match     | 1277/1319 (96.8158%) |  1281/1319 (97.1190%) | +4 (+0.3033 pp) |
| Flexible extract | 1277/1319 (96.8158%) |  1281/1319 (97.1190%) | +4 (+0.3033 pp) |

The isolated run shows no aggregate accuracy regression and an observed four-question improvement. The delta is smaller than one evaluation standard error, so this PR does not claim a statistically established accuracy gain. The correctness motivation is preserving the checkpoint’s declared FP32 values rather than relying on a BF16-to-FP32 upcast after precision has already been lost.

## AI assistance

OpenAI Codex assisted with analysis, implementation, validation, and drafting. Fangzhou Ai reviewed the complete source diff, authorized the DCO sign-off, and owns the submission.

---

## comments

