---
title: '55407'
pageLayout: default
slug: thoughts/inference/vllm-prs/55407
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/55407.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#55407](https://github.com/aarnphm/aarnphm.github.io/issues/55407) \[Bugfix\] Fix Kimi K3 NVFP4 MoE weight conversion OOM

[vllm-project/vllm#55407](https://github.com/vllm-project/vllm/pull/55407) · wzhao18 · MERGED · created 2026-09-04T21:46:34Z · merged 2026-09-07T03:45:26Z

## Purpose

Currently when running Kimi k3 nvfp4, it throws \~15000 lines of warning like the following due to fragmented memory during quantization post processing. In some cases, it also fatally fails with OOM.



```
[rank6]:[W904 12:06:06.083917542 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 6 while trying to allocate 20971520 bytes (free: 16252928, total: 287428640768).
[rank6]:[W904 12:06:06.087318728 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 6 while trying to allocate 20971520 bytes (free: 16252928, total: 287428640768).
```

This PR fixes it by optimizing nvfp4 MoE weight post processing memory usage. It preallocate the final FlashInfer TRT-LLM shuffled tensors and populate them one expert at a time. This avoids allocating tensors for per-expert result, which fails under `_scoped_allocator_max_split(max_split_size_mb=20)` as large blocks cannot be split up.

## Test Plan

## Test Result

---

<details>
<summary> Essential Elements of an Effective PR Description Checklist </summary>

- [ ] The purpose of the PR, such as “Fix some issue (link existing issues this PR will resolve)”.
- [ ] The test plan, such as providing test command.
- [ ] The test results, such as pasting the results comparison before and after, or e2e results
- [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.

</details>

---

## comments

**coderabbitai** (2026-09-04T21:47:41Z):

<!-- This is an auto-generated comment: summarize by coderabbit.ai -->

<!-- review_stack_entry_start -->

[![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/vllm-project/vllm/pull/55407)

<!-- review_stack_entry_end -->

<!-- walkthrough_start -->

<details>
<summary>📝 Summary</summary>

<!-- This is an auto-generated comment: release notes by coderabbit.ai -->

## Summary by CodeRabbit

- **Performance**
  - Improved FP4 mixture-of-experts weight preparation efficiency, reducing repeated processing and temporary allocations.
  - Reduced memory usage during model loading by releasing temporary accelerator cache before post-load transformations.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

## Walkthrough

The PR optimizes FP4 MoE weight shuffling with cached indices and preallocated buffers. Model loading now clears the accelerator cache before post-load weight transformations.

### Changes

**Model Loading Updates**

| Layer / File(s)                                                                                           | Summary                                                                                                                                                                |
| --------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **FP4 MoE weight preparation** <br> `vllm/model_executor/layers/quantization/utils/flashinfer_fp4_moe.py` | The preparation function caches permutation indices by tensor shape, writes results into preallocated buffers, and copies interleaved scale data into scratch buffers. |
| **Loader cache cleanup** <br> `vllm/model_executor/model_loader/base_loader.py`                           | `load_model` calls `torch.accelerator.empty_cache()` after weight loading and before post-load transformations.                                                        |

**Estimated code review effort:** 3 (Moderate) | \~20 minutes

<!-- final_review_risk_start -->

**Merge Risk:** _🟡 Moderate_ · up to `148cd`

<!-- final_review_risk_coverage:{"sourceCommitId":"148cd134a7b59c6e888dd9d13e11e99c2d0f69ea","coveredCommitId":"148cd134a7b59c6e888dd9d13e11e99c2d0f69ea","kind":"reviewed"} -->

HPU model loading can fail before post-load processing, so the cleanup path should be made backend-safe before merge.

<!-- final_review_risk_end -->

**Suggested reviewers:** `fxmarty-amd`

</details>

<!-- walkthrough_end -->

<!-- pre_merge_checks_walkthrough_start -->

<details>
<summary>🚥 Pre-merge checks | ✅ 5</summary>

<details>
<summary>✅ Passed checks (5 passed)</summary>

|         Check name         | Status   | Explanation                                                                                                                                                                                               |
| :------------------------: | :------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|     Docstring Coverage     | ✅ Passed | Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 2 files.                 |
|     Linked Issues check    | ✅ Passed | Check skipped because no linked issues were found for this pull request.                                                                                                                                  |
| Out of Scope Changes check | ✅ Passed | Check skipped because no linked issues were found for this pull request.                                                                                                                                  |
|         Title check        | ✅ Passed | The title clearly identifies the bugfix and the affected Kimi K3 NVFP4 MoE weight conversion path. It matches the main changes that reduce memory use and prevent OOM failures.                           |
|      Description check     | ✅ Passed | The description explains the fragmented-memory warnings and OOM failures, identifies the NVFP4 MoE post-processing optimization, and describes the preallocation approach. The empty test plan and test … |

</details>

</details>

<!-- pre_merge_checks_walkthrough_end -->

<!-- tips_start -->

---

Thanks for using [CodeRabbit](https://coderabbit.ai?utm_source=oss\&utm_medium=github\&utm_campaign=vllm-project/vllm\&utm_content=55407)! It’s free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

<details>
<summary>❤️ Share</summary>

- [X](https://twitter.com/intent/tweet?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A\&url=https%3A//coderabbit.ai)
- [Mastodon](https://mastodon.social/share?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A%20https%3A%2F%2Fcoderabbit.ai)
- [Reddit](https://www.reddit.com/submit?title=Great%20tool%20for%20code%20review%20-%20CodeRabbit\&text=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code.%20Check%20it%20out%3A%20https%3A//coderabbit.ai)
- [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcoderabbit.ai\&mini=true\&title=Great%20tool%20for%20code%20review%20-%20CodeRabbit\&summary=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code)

</details>

<sub>Comment `@coderabbitai help` to get the list of available commands.</sub>

<!-- tips_end -->

**mgoin** (2026-09-05T02:59:27Z):

/ci run

**github-actions** (2026-09-05T02:59:45Z):

✅ Triggered [Buildkite CI #87358](https://buildkite.com/vllm/ci/builds/87358) for commit `148cd134a7b5`.

**wzhao18** (2026-09-05T04:49:29Z):

/ci retry

**github-actions** (2026-09-05T04:49:47Z):

✅ Queued 3 failed job(s) for retry in [Buildkite CI #87358](https://buildkite.com/vllm/ci/builds/87358).

**wzhao18** (2026-09-05T04:50:29Z):

/ci cancel

**github-actions** (2026-09-05T04:51:14Z):

✅ Requested cancellation of 1 CI build for `wzhao/fix-k3-nvfp4-load`: [#87358](https://buildkite.com/vllm/ci/builds/87358).

**wzhao18** (2026-09-05T05:08:24Z):

/ci run

**github-actions** (2026-09-05T05:08:44Z):

✅ Triggered [Buildkite CI #87373](https://buildkite.com/vllm/ci/builds/87373) for commit `2d3f78a47b64`.

