---
title: '55242'
pageLayout: default
slug: thoughts/inference/vllm-prs/55242
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/55242.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#55242](https://github.com/aarnphm/aarnphm.github.io/issues/55242) \[Perf\] Kimi K3 nvfp4 Align in_proj weights by 128 to avoid elementwise copy

[vllm-project/vllm#55242](https://github.com/vllm-project/vllm/pull/55242) · wzhao18 · MERGED · created 2026-09-03T21:42:19Z · merged 2026-09-05T02:46:47Z

## Purpose

This PR updates the following:

- Align Kimi K3 nvfp4 in_proj weights by 128 to avoid elementwise copy from `.contiguous()` to make the output compact
- Update modelopt Fp8_Pb_Wo to select linear method backend based on padded weight layout (instead of original weight), before this PR it will reject DeepGEMM and chooses Cutlass due to misalignment with DeepGEMM’s requirement.
- Fix GEMM-AR fusion disabled warning to only warn once instead of for every layer.

Before this PR (Cutlass fp8 GEMM + elementwise copy)
<img width="807" height="162" alt="Screenshot 2026-09-03 at 5 41 10 PM" src="https://github.com/user-attachments/assets/bb7cc1cc-4f8f-41e4-b359-615299f7492b" />

After this PR (DeepGEMM fp8 GEMM with the following elementwise copy removed)
<img width="781" height="135" alt="Screenshot 2026-09-03 at 5 41 32 PM" src="https://github.com/user-attachments/assets/c5c042ba-a903-4e4c-bf67-434c3b4283b6" />

## Test Plan

## Test Result

---

<details>
<summary> Essential Elements of an Effective PR Description Checklist </summary>

- [ ] The purpose of the PR, such as “Fix some issue (link existing issues this PR will resolve)”.
- [ ] The test plan, such as providing test command.
- [ ] The test results, such as pasting the results comparison before and after, or e2e results
- [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.

</details>

---

## comments

**coderabbitai** (2026-09-03T21:43:14Z):

<!-- This is an auto-generated comment: summarize by coderabbit.ai -->

<!-- review_stack_entry_start -->

[![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/vllm-project/vllm/pull/55242)

<!-- review_stack_entry_end -->

<!-- walkthrough_start -->

<details>
<summary>📝 Summary</summary>

<!-- This is an auto-generated comment: release notes by coderabbit.ai -->

## Summary by CodeRabbit

- **Bug Fixes**
  - Improved FP8 quantization handling for Kimi K3 models by using the correct padded projection dimensions, helping ensure compatible kernel initialization.
  - Updated projection alignment behavior for FP8 blockwise quantization.
  - Standardized warnings when optimized GEMM paths are unavailable, clearly indicating that execution falls back to an unfused path.
- **Tests**
  - Updated quantization coverage to validate padded kernel dimensions and compatibility with revised kernel selection behavior.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

## Walkthrough

The change passes format-specific weight shapes to ModelOpt FP8 kernels, applies FP8_PB_WO alignment in Kimi K3, and centralizes GEMM-RS/AR incompatibility warnings.

### Changes

**ModelOpt and Kimi K3 quantization updates**

| Layer / File(s)                                                                                                                                                                                                       | Summary                                                                                                                                                                                        |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **ModelOpt kernel weight-shape dispatch** <br> `vllm/model_executor/layers/quantization/modelopt.py`, `tests/quantization/test_modelopt.py`                                                                           | ModelOpt adds a kernel weight-shape hook. FP8 partial-block formats return 128-aligned shapes. Kernel initialization receives the selected shape. Tests update mocks and verify `(2688, 128)`. |
| **Kimi K3 FP8 projection alignment** <br> `vllm/models/kimi_k3/nvidia/kda.py`                                                                                                                                         | The `in_proj_qkvgfab` projection uses 128 alignment for `FP8_PB_WO` and 16 alignment otherwise.                                                                                                |
| **GEMM-RS/AR incompatibility warnings** <br> `vllm/models/kimi_k3/nvidia/ops/cute_dsl/gemm_rs_ar.py`, `vllm/models/kimi_k3/nvidia/kda.py`, `vllm/models/kimi_k3/nvidia/mla.py`, `vllm/models/kimi_k3/nvidia/model.py` | `GemmRsAr.warn_incompatible_projection()` emits the one-time warning. Kimi K3 callers use this method for incompatible projections.                                                            |

**Estimated code review effort:** 3 (Moderate) | \~20 minutes

<!-- final_review_risk_start -->

**Merge Risk:** _🔵 Low_ · up to `80a12`

<!-- final_review_risk_coverage:{"sourceCommitId":"80a12d774f0b0bf9e567afd6b832374decf8e37f","coveredCommitId":"80a12d774f0b0bf9e567afd6b832374decf8e37f","kind":"reviewed"} -->

Kimi K3 deployments using standalone FP8_PB_WO quantization may retain the avoidable padded-weight copy that this change is intended to remove, reducing inference performance for those configurations. Resolve the standalone configuration handling before merge.

<!-- final_review_risk_end -->

**Suggested reviewers:** `gau-nernst`

</details>

<!-- walkthrough_end -->

<!-- pre_merge_checks_walkthrough_start -->

<details>
<summary>🚥 Pre-merge checks | ✅ 4 | ❌ 1</summary>

### ❌ Failed checks (1 warning)

|     Check name     | Status     | Explanation                                                                                                                                                                                 | Resolution                                                                         |
| :----------------: | :--------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :--------------------------------------------------------------------------------- |
| Docstring Coverage | ⚠️ Warning | Docstring coverage is 36.84% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 19 functions across 6 files. | Write docstrings for the functions missing them to satisfy the coverage threshold. |

<details>
<summary>✅ Passed checks (4 passed)</summary>

|         Check name         | Status   | Explanation                                                                                                                                                                                       |
| :------------------------: | :------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|     Linked Issues check    | ✅ Passed | Check skipped because no linked issues were found for this pull request.                                                                                                                          |
| Out of Scope Changes check | ✅ Passed | Check skipped because no linked issues were found for this pull request.                                                                                                                          |
|         Title check        | ✅ Passed | The title clearly identifies the primary Kimi K3 nvfp4 alignment change and its purpose of avoiding an elementwise copy.                                                                          |
|      Description check     | ✅ Passed | The description accurately covers the weight alignment, ModelOpt FP8 backend selection, and GEMM-AR warning changes. The empty test sections do not prevent a pass because this check is lenient. |

</details>

</details>

<!-- pre_merge_checks_walkthrough_end -->

- [ ] <!-- {"checkboxId":"585bb3f6-faf5-4dbf-96d2-74e382adf19a"} --> Fix all pre-merge checks with AI

<!-- tips_start -->

---

Thanks for using [CodeRabbit](https://coderabbit.ai?utm_source=oss\&utm_medium=github\&utm_campaign=vllm-project/vllm\&utm_content=55242)! It’s free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

<details>
<summary>❤️ Share</summary>

- [X](https://twitter.com/intent/tweet?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A\&url=https%3A//coderabbit.ai)
- [Mastodon](https://mastodon.social/share?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A%20https%3A%2F%2Fcoderabbit.ai)
- [Reddit](https://www.reddit.com/submit?title=Great%20tool%20for%20code%20review%20-%20CodeRabbit\&text=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code.%20Check%20it%20out%3A%20https%3A//coderabbit.ai)
- [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcoderabbit.ai\&mini=true\&title=Great%20tool%20for%20code%20review%20-%20CodeRabbit\&summary=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code)

</details>

<sub>Comment `@coderabbitai help` to get the list of available commands.</sub>

<!-- tips_end -->

**wzhao18** (2026-09-04T20:23:22Z):

/ci run

**github-actions** (2026-09-04T20:23:41Z):

✅ Triggered [Buildkite CI #87315](https://buildkite.com/vllm/ci/builds/87315) for commit `b1e37f0d5da9`.

**wzhao18** (2026-09-04T21:56:53Z):

/ci retry

**github-actions** (2026-09-04T21:57:12Z):

✅ Queued 2 failed job(s) for retry in [Buildkite CI #87315](https://buildkite.com/vllm/ci/builds/87315).

