---
title: '50484'
pageLayout: default
slug: thoughts/inference/vllm-prs/50484
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/50484.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#50484](https://github.com/aarnphm/aarnphm.github.io/issues/50484) \[Kimi-K3\] DCP support

[vllm-project/vllm#50484](https://github.com/vllm-project/vllm/pull/50484) · GirasoleY · MERGED · created 2026-07-30T21:59:28Z · merged 2026-08-10T19:54:27Z

Co-authored-by: @foraxe

## Summary

Follow-up to [#50000](https://github.com/aarnphm/aarnphm.github.io/issues/50000) adding decode context parallelism for Kimi-K3:

- direct symmetric-memory DCP A2A output/LSE reduction, including empty-KV-shard masking
- DCP support for the fused Kimi-K3 MLA layer
- NVLS-multicast direct query gather and multimem chunked-context KV gather
- direct publication of query shards into the consumer-final buffer, removing the staging-to-final materialization by @foraxe

This replaces [#50055](https://github.com/aarnphm/aarnphm.github.io/issues/50055).

## Query-buffer lifetime

The direct query gather returns a view of its persistent symmetric buffer. Reuse is ordered by the mandatory post-attention DCP combine:

1. query gather publishes the current epoch and waits for every rank
2. attention consumes the gathered query on the same CUDA stream
3. every combine implementation introduces a cross-rank dependency after attention
4. the next gather cannot execute until all ranks have passed the preceding attention read

DBO microbatches use independent workspace slots.

## Accuracy

`.buildkite/kimi-k3/rack5/gsm8k.yaml`:

| Arm  | exact_match | Correct/N | Request errors |
| ---- | ----------: | --------: | -------------: |
| TP8  |      96.21% | 1269/1319 |              0 |
| DCP8 |      96.97% | 1279/1319 |              0 |

## Performance

120k regression workload: 114k shared prefix + 6k suffix, 400 output tokens, Mooncake disabled.

| Arm  | Conc |       TPOT p50/p90/p99 | HBM use mean/peak |
| ---- | ---: | ---------------------: | ----------------: |
| TP8  |   c1 | 13.806/13.831/13.857ms |       4.93%/7.17% |
| DCP8 |   c1 | 10.544/10.574/10.597ms |       0.64%/1.10% |
| TP8  |   c2 | 16.192/23.591/34.101ms |      9.18%/14.11% |
| DCP8 |   c2 | 11.828/19.437/27.562ms |       1.30%/2.37% |

HBM KV capacity per GPU:

- TP8: 1.933M tokens / 25.12 GiB
- DCP8: 19.752M tokens / 34.28 GiB

On 4x GB200 at matched Kimi-K3 geometry, the direct-to-final query path improved query-exchange latency by 10.4%-29.9%. Correctness and changing-input CUDA-graph replay passed.

## Scope and overlap

This PR subsumes the direct symmetric-memory MLA A2A work in [#48897](https://github.com/aarnphm/aarnphm.github.io/issues/48897) and extends it with Kimi-K3 fused MLA DCP, direct query gather, chunked-context KV gather, and the direct-to-final query optimization.

## CUDA-graph padding follow-up

Commit `76b2e9d45e` makes empty-shard LSE masking tolerate CUDA-graph padding rows.

Validation:

- `tests/distributed/test_dcp_a2a.py::TestLSEWeightedCombine::test_ag_rs_masks_empty_shard_and_padded_lse`: 1 passed on GB300.
- DCP2 partial-prefix reuse versus TP2 on 2x GB300, with direct A2A, query gather, and KV gather selected: 1 passed; 256 cached tokens, exact generated-token match, finite logprobs, and drift <span>&lArr;</span> 1e-2.
- Pre-commit hooks on both changed files: passed.
- An exploratory full-file GB300 run had 25 passes and 7 return-LSE dtype assertion failures (float32 actual versus fp16/bf16 expected); those cases pass no masking metadata and do not execute this change.

AI assistance was used to diagnose, implement, and validate this follow-up fix.

---

## comments

**mergify** (2026-07-31T13:54:40Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @GirasoleY.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**prakhar-prakash-juspay** (2026-08-04T11:02:14Z):

@GirasoleY do you plan to get this PR merged if it’s working fine?

**github-actions** (2026-08-04T21:19:24Z):

✅ @GirasoleY, CI is now available for this PR. Comment `/ci run` to run full CI or `/ci retry` to retry failed jobs.

<!-- vllm-ci-authorized -->

**prakhar-prakash-juspay** (2026-08-06T03:49:46Z):

Hey would this work with the dspark speculative model?

**mergify** (2026-08-06T21:59:28Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @GirasoleY.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**WoosukKwon** (2026-08-07T01:52:02Z):

/ci run

**github-actions** (2026-08-07T01:52:19Z):

✅ Triggered [Buildkite CI #82764](https://buildkite.com/vllm/ci/builds/82764) for commit `76b2e9d45eee`.

**WoosukKwon** (2026-08-07T23:21:01Z):

/ci run

**github-actions** (2026-08-07T23:21:17Z):

✅ Triggered [Buildkite CI #82945](https://buildkite.com/vllm/ci/builds/82945) for commit `65328d0ee326`.

**WoosukKwon** (2026-08-10T16:32:17Z):

/ci run

**github-actions** (2026-08-10T16:32:34Z):

✅ Triggered [Buildkite CI #83171](https://buildkite.com/vllm/ci/builds/83171) for commit `4b4c1d0214d9`.

