---
title: '53327'
pageLayout: default
slug: thoughts/inference/vllm-prs/53327
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/53327.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#53327](https://github.com/aarnphm/aarnphm.github.io/issues/53327) \[Bugfix\]\[Kimi K3\] Enable deferred MoE finalization before weight loading

[vllm-project/vllm#53327](https://github.com/vllm-project/vllm/pull/53327) · zyongye · MERGED · created 2026-08-21T21:59:32Z · merged 2026-08-22T03:23:33Z

## Summary

- Enable K3 deferred MoE finalization from the selected monolithic expert class during runner construction.
- Keep the existing hidden-size and parallel-topology guards.
- Log when deferred top-k finalization is active and add a regression test for the pre-weight-loading lifecycle.

## Root cause

[#53152](https://github.com/aarnphm/aarnphm.github.io/issues/53152) checked `quant_method.moe_kernel` in `LatentMoERunner.__init__`. `Mxfp4MoEMethod` initializes that field to `None` and only creates the kernel later in `process_weights_after_loading`, so the deferred path was always disabled during runner construction even when the supported FlashInfer TRTLLM MXFP4 expert class had already been selected.

The tail operator must know whether it consumes an `UnfinalizedMoEOutput` during construction so its CuTeDSL warmup units are registered with the correct top-k shape. This change therefore uses the already-selected expert class at that lifecycle point.

## Duplicate-work check

- This is a follow-up bug fix to merged PR [#53152](https://github.com/aarnphm/aarnphm.github.io/issues/53152) and belongs to [#50587](https://github.com/aarnphm/aarnphm.github.io/issues/50587).
- Searched open PRs for `53152`, `50587`, `K3 latent MoE deferred finalize`, and `latent moe tail fusion`.
- Open PR [#53310](https://github.com/aarnphm/aarnphm.github.io/issues/53310) also follows [#53152](https://github.com/aarnphm/aarnphm.github.io/issues/53152), but it changes `UnfinalizedMoEOutput` typing in other runners and does not touch this initialization-order bug.
- No open PR addresses this fix.

## Tests



```bash
.venv/bin/python -m pytest \
  tests/models/kimi_k3/test_latent_moe_tail.py::test_deferred_finalize_enabled_before_moe_kernel_setup \
  tests/models/kimi_k3/test_latent_moe_tail.py::test_latent_moe_tail_deferred_finalize_matches_finalized \
  -q
```

Result: `2 passed` on 8x NVIDIA B300. The second test compares deferred and finalized outputs across TP8.



```bash
.venv/bin/pre-commit run ruff-check --files \
  vllm/models/kimi_k3/nvidia/latent_moe_runner.py \
  tests/models/kimi_k3/test_latent_moe_tail.py
.venv/bin/pre-commit run ruff-format --files \
  vllm/models/kimi_k3/nvidia/latent_moe_runner.py \
  tests/models/kimi_k3/test_latent_moe_tail.py
```

Result: both passed.

Serving validation: Kimi-K3 TP8 selected the FlashInfer TRTLLM MXFP4/MXFP8 backend, reported deferred top-k finalization enabled, and completed 40/40 requests with no failures.

## AI assistance

OpenAI Codex was used to diagnose the initialization-order issue, draft the code and test, run validation, and prepare this PR. The human submitter must review every changed line and understand and defend the change end-to-end.

---

## comments

**zyongye** (2026-08-21T22:16:31Z):

/ci run

**github-actions** (2026-08-21T22:16:49Z):

✅ Triggered [Buildkite CI #85122](https://buildkite.com/vllm/ci/builds/85122) for commit `bbbf7405d192`.

