---
title: '50000'
pageLayout: default
slug: thoughts/inference/vllm-prs/50000
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/50000.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#50000](https://github.com/aarnphm/aarnphm.github.io/issues/50000) \[New model\] Kimi K3

[vllm-project/vllm#50000](https://github.com/vllm-project/vllm/pull/50000) · ZJY0516 · MERGED · created 2026-07-27T15:19:13Z · merged 2026-07-30T10:49:58Z

## Purpose

**7.30 Note: after merging, it still needs to install <https://github.com/flashinfer-ai/flashinfer/releases/tag/v0.6.16rc5> to run K3**

add `moonshotai/Kimi-K3` model support

Please use the docker image `vllm/vllm-openai:kimi-k3` to run it, see vllm recipes <https://recipes.vllm.ai/moonshotai/Kimi-K3>.

This branch has some extra dependencies:

- SITU trtllmgen MOE: [flashinfer-ai/flashinfer#4180](https://github.com/flashinfer-ai/flashinfer/pull/4180)
- MLA: [flashinfer-ai/flashinfer#4178](https://github.com/flashinfer-ai/flashinfer/pull/4178)

See our blog post <https://vllm.ai/blog/2026-07-27-k3> for optimization details

### Spec decoding

Dspark: <https://huggingface.co/Inferact/Kimi-K3-DSpark>

## How to merge this

We split it into several PRs

kernels and model files: [vllm-project/vllm#50089](https://github.com/vllm-project/vllm/pull/50089) [vllm-project/vllm#50090](https://github.com/vllm-project/vllm/pull/50090)
python front end: [vllm-project/vllm#50093](https://github.com/vllm-project/vllm/pull/50093)
rust frontend: [vllm-project/vllm#50104](https://github.com/vllm-project/vllm/pull/50104)

---

<details>
<summary> Essential Elements of an Effective PR Description Checklist </summary>

- [ ] The purpose of the PR, such as “Fix some issue (link existing issues this PR will resolve)”.
- [ ] The test plan, such as providing test command.
- [ ] The test results, such as pasting the results comparison before and after, or e2e results
- [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.

</details>

---

## comments

**mergify** (2026-07-27T15:20:14Z):

Documentation preview: [https://vllm—50000.org.readthedocs.build/en/50000/](https://vllm--50000.org.readthedocs.build/en/50000/)

**mergify** (2026-07-27T15:20:36Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @ZJY0516.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**tlrmchlsmth** (2026-07-27T15:20:51Z):

50k get!

**abmfy** (2026-07-27T15:24:18Z):

Congrats on K3 && 50k-th PR! 🎉

**robertgshaw2-redhat** (2026-07-27T15:38:49Z):

added ready label to kick off container build

**jeejeelee** (2026-07-27T15:48:26Z):

> added ready label to kick off container build

also see `vllm/vllm-openai:kimi-k3`

**tjtanaa** (2026-07-27T23:12:37Z):

Congratz!!!!

**aoshen02** (2026-07-28T04:19:08Z):

RL is supported in the rollout side: [vllm-project/vllm#50079](https://github.com/vllm-project/vllm/issues/50079)

**mergify** (2026-07-28T04:27:41Z):

Hi @ZJY0516, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

**tjtanaa** (2026-07-28T04:52:15Z):

I installed from this branch, running using the command from <https://recipes.vllm.ai/moonshotai/Kimi-K3?hardware=mi355x> and I get this error.



```
(Worker_TP1 pid=6338) ERROR 07-28 04:05:35 [multiproc_executor.py:912]                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=6338) ERROR 07-28 04:05:35 [multiproc_executor.py:912]   File "/app/reviewpr06-3/kimi-k3/vllm/model_executor/layers/fused_moe/oracle/mxfp4.py", line 630, in select_deepseek_v4_mxfp4_moe_backend
(Worker_TP1 pid=6338) ERROR 07-28 04:05:35 [multiproc_executor.py:912]     raise NotImplementedError(
(Worker_TP1 pid=6338) ERROR 07-28 04:05:35 [multiproc_executor.py:912] NotImplementedError: No MXFP4 MoE backend supports the deployment configuration.
```

This issue is mitigated if we are using the Day-0 docker image `vllm/vllm-openai-rocm:kimi-k3` as base.

@hongxiayang can you share the details of the docker image environment necessary to run kimi k3?

CC @dllehr-amd @AndreasKaratzas @Rohan138 as we will need to coordinate the upgrades. especially the aiter version.

**bbartels** (2026-07-28T12:58:50Z):

Could we get a new kimi image published? the current one seems to have bugs in tool calling that are fixed on newer commits on this pr!

**varungup90** (2026-07-28T15:46:24Z):

> Could we get a new kimi image published? the current one seems to have bugs in tool calling that are fixed on newer commits on this pr!

please help release image for cu-129 also. Thank you.

**hongxiayang** (2026-07-28T17:47:28Z):

> I installed from this branch, running using the command from <https://recipes.vllm.ai/moonshotai/Kimi-K3?hardware=mi355x> and I get this error.
>
>
>
> ```
> (Worker_TP1 pid=6338) ERROR 07-28 04:05:35 [multiproc_executor.py:912]                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> (Worker_TP1 pid=6338) ERROR 07-28 04:05:35 [multiproc_executor.py:912]   File "/app/reviewpr06-3/kimi-k3/vllm/model_executor/layers/fused_moe/oracle/mxfp4.py", line 630, in select_deepseek_v4_mxfp4_moe_backend
> (Worker_TP1 pid=6338) ERROR 07-28 04:05:35 [multiproc_executor.py:912]     raise NotImplementedError(
> (Worker_TP1 pid=6338) ERROR 07-28 04:05:35 [multiproc_executor.py:912] NotImplementedError: No MXFP4 MoE backend supports the deployment configuration.
> ```
>
> This issue is mitigated if we are using the Day-0 docker image `vllm/vllm-openai-rocm:kimi-k3` as base.
>
> @hongxiayang can you share the details of the docker image environment necessary to run kimi k3?
>
> CC @dllehr-amd @AndreasKaratzas @Rohan138 as we will need to coordinate the upgrades. especially the aiter version.

with day 0 docker image is a good start.

There is aiter release tag to prepare upgrade. let’s discuss internally about release details.

**mergify** (2026-07-29T11:10:25Z):

Hi @ZJY0516, the pre-commit checks have failed. Please run:



```bash
uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files
```

Then, commit the changes and push to your branch.

For future commits, `pre-commit` will run automatically on changed files before each commit.

**mergify** (2026-07-29T15:51:40Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @ZJY0516.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**ZJY0516** (2026-07-30T13:49:23Z):

Note: after merging, it still needs to install <https://github.com/flashinfer-ai/flashinfer/releases/tag/v0.6.16rc5> to run K3

