---
title: '53396'
pageLayout: default
slug: thoughts/inference/vllm-prs/53396
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/53396.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#53396](https://github.com/aarnphm/aarnphm.github.io/issues/53396) \[Kimi K3\]\[Kernel\] Support DS conv-state layout in fused KDA decode kernel

[vllm-project/vllm#53396](https://github.com/vllm-project/vllm/pull/53396) · gcanlin · MERGED · created 2026-08-22T19:17:12Z · merged 2026-08-27T06:54:51Z

## Summary

The fused KDA decode kernel (`ops.fused_kda_decode`) previously required the SD conv-state cache layout, so any deployment that pins `VLLM_SSM_CONV_STATE_LAYOUT=DS` — required for NIXL point-to-point state transfer in P/D-disaggregated serving — silently bypassed the fused path and fell back to the multi-op Triton decode chain for all 69 KDA layers of the model.

This PR supports both physical cache layouts by dispatching to separate SD- and DS-specialized fused-kernel instantiations. Keeping each layout’s address calculation compile-time constant avoids the extra integer instructions and register pressure caused by runtime inner strides. Both layouts now use the fused path (in-proj GEMM → fused KDA decode → out-proj GEMM per layer and step), and numerics are bit-identical for identical inputs.

Update (head `0e67480f`): per review feedback, the layout specialization is now expressed as two integer conv-state inner strides (channel, tap) as non-type template parameters instead of a layout enum. The generated device code is unchanged — see the parity data below.

## Motivation

- Deployment reality: production K3 serving pins `VLLM_SSM_CONV_STATE_LAYOUT=DS` (NIXL PD transfer), and several recent fixes treat DS as the supported production layout. Under DS, the fused kernel was never used.
- Measured cost of the fallback chain on B200 (TP8 shape, H_local=12, K=V=128): 1.16-1.58x slower than the fused kernel depending on batch, plus \~3x launch count in the per-layer decode chain (8-9 vs 3 kernels).

## Performance

### DS specialized fused path vs Triton fallback

B200, CUDA graph replay, `H_local=12`, `K=V=128`, 50 warmup iterations, and 300 measured iterations. Values are the average of two median runs. The fallback uses the Triton gated RMSNorm kernel rather than its eager PyTorch decomposition.

| Decode batch size | Triton fallback (us) | Specialized fused (us) | Speedup |
| ----------------: | -------------------: | ---------------------: | ------: |
|                 1 |                9.170 |                  5.500 |   1.67x |
|                 8 |               10.480 |                  5.610 |   1.87x |
|                32 |               16.865 |                  9.170 |   1.84x |
|                64 |               24.920 |                 16.000 |   1.56x |
|               128 |               50.625 |                 35.195 |   1.44x |
|               256 |               90.890 |                 71.075 |   1.28x |
|               512 |              171.455 |                140.055 |   1.22x |

### SD regression and DS specialization check

B200, CUDA graph replay, `H_local=12`, `K=V=128`, 50 warmup iterations, and 300 measured iterations. Values are the average of two median runs with reversed execution order, using binaries produced by the same build tree.

|  Decode batch size | SD baseline | SD dynamic strides | SD regression | SD specialized | SD vs. baseline | DS dynamic strides | DS specialized | DS improvement |
| -----------------: | ----------: | -----------------: | ------------: | -------------: | --------------: | -----------------: | -------------: | -------------: |
|                  1 |    5.250 us |           5.400 us |        +2.86% |       5.250 us |           0.00% |           5.530 us |       5.490 us |          0.72% |
|                  8 |    5.340 us |           5.460 us |        +2.25% |       5.340 us |           0.00% |           5.790 us |       5.610 us |          3.11% |
|                 32 |    8.850 us |           8.960 us |        +1.24% |       8.850 us |           0.00% |           9.285 us |       9.170 us |          1.24% |
|                 64 |   15.670 us |          16.475 us |        +5.14% |      15.670 us |           0.00% |          16.815 us |      16.000 us |          4.85% |
|                128 |   35.570 us |          39.670 us |       +11.53% |      35.570 us |           0.00% |          39.515 us |      35.190 us |         10.95% |
|                256 |   71.675 us |          76.245 us |        +6.38% |      71.650 us |          -0.03% |          75.905 us |      71.150 us |          6.26% |
|                512 |  140.925 us |         149.695 us |        +6.22% |     140.945 us |          +0.01% |         148.600 us |     140.030 us |          5.77% |
| **Geometric mean** |           — |                  — |    **+5.04%** |              — |      **-0.00%** |                  — |              — |      **4.75%** |

Compile-time specialization fully recovers the original SD performance and improves DS performance by up to **10.95%**. For the representative H_local=12 kernel, runtime strides increase register use from 64 to 72 registers per thread, while both specialized variants use 64. The additional kernel instances increase the extension binary by 392 KB (0.95%).

### Stride-NTTP parity check (review feedback, head `0e67480f`)

The layout enum was replaced by two integer strides as template parameters, per review feedback. The layout is still detected at launch from the conv_state strides and resolved inside the head-count dispatch (SD = `<1, 3 * H * 128>`, DS = `<kConvWidth - 1, 1>`), so both strides remain compile-time constants.

Codegen parity proof: a normalized `cuobjdump` comparison of all 128 `kda_decode_fusion_many_heads_kernel` instantiations in `_C_stable_libtorch` shows **bit-identical SASS** between the enum version and this rewrite (identical REG/STACK/SHARED, zero spills), so measured performance is identical by construction.

B200, CUDA graph replay, `H_local=12`, `K=V=128`, 50 warmup / 300 measured iterations, fused-path per-layer latency:

| Decode batch size | SD enum (us) | SD stride-NTTP (us) |  Delta | DS enum (us) | DS stride-NTTP (us) |  Delta |
| ----------------: | -----------: | ------------------: | -----: | -----------: | ------------------: | -----: |
|                 1 |         5.02 |                4.99 |  -0.6% |         5.25 |                5.25 |   0.0% |
|                 8 |         5.31 |                5.31 |   0.0% |         5.58 |                5.58 |   0.0% |
|                32 |         8.82 |                8.82 |   0.0% |         9.16 |                9.16 |   0.0% |
|                64 |        15.64 |               15.66 |  +0.1% |        15.98 |               15.98 |   0.0% |
|               128 |        35.38 |               35.37 | -0.03% |        35.07 |               35.04 | -0.09% |

Production-shaped check (69 per-layer state buffer sets, B=128, DS): 37.04 us/layer fused vs 49.33 us/layer Triton chain (1.33x).

An earlier stride-NTTP attempt was measured to regress 7-18%, but that attempt bundled unrelated changes; this clean rewrite matches the enum version exactly.

### E2E: 2 nodes x 8xB200, TP8+PP2, FP8 KV cache, DS layout, `FULL_DECODE_ONLY` cudagraphs

`vllm bench serve`, random dataset, 128 input / 512 output tokens,
128 requests, max-concurrency 64, temperature 0, and no profiler. Each side used
one full-load pilot followed by two measured runs. Both sides used the same vLLM
nightly image (`sha256:95bed119f39e...`); the candidate overlaid this PR at
commit `40459de9d3`. The current head `0e67480f` emits bit-identical kernel SASS (verified above), so these numbers carry over; a fresh E2E run on it will follow.

| Metric                 | Baseline (fallback chain) | This PR (fused kernel) |     Change |
| ---------------------- | ------------------------: | ---------------------: | ---------: |
| Output throughput      |             2069.73 tok/s |          2149.26 tok/s | **+3.84%** |
| Total token throughput |             2587.16 tok/s |          2686.57 tok/s | **+3.84%** |
| Benchmark duration     |                   31.66 s |                30.49 s | **-3.70%** |
| Mean TPOT              |                  29.73 ms |               28.59 ms | **-3.84%** |
| Median TPOT            |                  29.99 ms |               28.77 ms | **-4.08%** |
| Mean ITL               |                  29.68 ms |               28.54 ms | **-3.84%** |
| Mean TTFT              |                 632.44 ms |              629.93 ms |     -0.40% |
| Median TTFT            |                 614.08 ms |              614.34 ms |     +0.04% |

Every measured run completed all 128 requests with zero failures. The
output-throughput spread was 0.46% for the baseline and 0.53% for this PR, so
the 3.84% improvement is well above the observed run-to-run variation. Both
nodes logged `Fused KDA decode kernel (conv+KDA+norm) is enabled.` for the
candidate, with no errors or runtime JIT-compilation warnings.

Acc:



```
| Tasks | Version | Filter           | n-shot | Metric       |   |  Value |   |     Stderr |
| ----- | ------: | ---------------- | -----: | ------------ | - | -----: | - | ---------: |
| gsm8k |       3 | flexible-extract |      5 | exact\_match | ↑ | 0.9689 | ± |     0.0048 |
|       |         | strict-match     |      5 | exact\_match | ↑ | 0.9697 | ± |     0.0047 |
```

### Kernel trace verification

With the DS layout pinned, the patched serving stack runs `kda_decode_fusion_many_heads_kernel` for every KDA decode layer (5076 calls in the profiled window, 7.2us/call average, 1.26% of window kernel time); the decode-side `_causal_conv1d_update_kernel`, `fused_recurrent_kda_packed_decode` triton kernels and the separate gated-norm kernel disappear from the decode hot path (prefill-only `_causal_conv1d_fwd_kernel` and FlashKDA prefill remain, as intended). In the baseline trace the fallback trio costs \~3.5% of kernel time.

## Test plan



```
pytest tests/models/kimi_k3/test_kda.py -k test_fused_kda_decode_correctness -v
```

`10 passed, 51 deselected`; the targeted cases cover both SD and DS layouts, H=12/24/48/96, output normalization, and lower-bound decay. Re-run at head `0e67480f` (stride-NTTP rewrite): `10 passed, 51 deselected`.

Server (as used for the E2E numbers above):



```bash
# on two 8xB200 nodes (rank0/rank1)
VLLM_SSM_CONV_STATE_LAYOUT=DS vllm serve /path/to/Kimi-K3 \
  --served-model-name Kimi-K3 --trust-remote-code \
  --max-model-len 1048576 --gpu-memory-utilization 0.84 \
  --kv-cache-dtype fp8 \
  --tensor-parallel-size 8 --pipeline-parallel-size 2 \
  --nnodes 2 --node-rank {0,1} --master-addr <rank0-ip> --master-port 29512 \
  --max-num-seqs 128 --max-num-batched-tokens 32768 \
  --max-cudagraph-capture-size 128 \
  --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}'
```

---

## comments

**mergify** (2026-08-22T19:36:28Z):

Documentation preview: [https://vllm—53396.org.readthedocs.build/en/53396/](https://vllm--53396.org.readthedocs.build/en/53396/)

**gau-nernst** (2026-08-25T09:48:27Z):

Do you have microbenchmark results to see if this change causes any regression to the current SD codepath?

**gcanlin** (2026-08-25T12:51:12Z):

> Do you have microbenchmark results to see if this change causes any regression to the current SD codepath?

Oh, it indeed causes regression in small batch, but not sure why have improvement for batch 256, 512.



```
| Batch | main SD | this PR SD | Delta |
|------:|---------:|-----------:|-----:|
| 1     | 5.22 µs  | 5.31 µs    | +1.7% |
| 8     | 5.31 µs  | 5.40 µs    | +1.7% |
| 32    | 8.73 µs  | 8.84 µs    | +1.3% |
| 64    | 15.63 µs | 16.38 µs   | +4.8% |
| 128   | 37.99 µs | 38.69 µs   | +1.8% |
| 256   | 77.42 µs | 74.29 µs   | -4.0% |
| 512   | 151.63 µs | 146.55 µs | -3.4% |
```

The root cause may be that this PR changes `cs_q_for_slot[hk + w * kPackedDim];` to `cs_q_for_slot[hk * conv_channel_stride + w * conv_tap_stride];`. I plan to use template to avoid add the computation of the original way. What do you think?

**gau-nernst** (2026-08-25T12:55:25Z):

yea the strides can be constexpr, should be fine

**gcanlin** (2026-08-25T13:57:16Z):

@gau-nernst Thanks for suggestion! Now it doesn’t cause any regression for SD layout and the new implementation is better than the old one for DS layout(I have updated the new data in PR description), so I run E2E benchmark again to see the benefits.

@tjtanaa @AndreasKaratzas Could you please take a look for the ROCm part? If it’s not suitable for ROCm, I can revert it.

**gau-nernst** (2026-08-27T03:20:57Z):

/ci run

**github-actions** (2026-08-27T03:21:15Z):

✅ Triggered [Buildkite CI #85738](https://buildkite.com/vllm/ci/builds/85738) for commit `0e67480f55e3`.

**github-actions** (2026-08-27T03:32:31Z):

✅ @gcanlin, CI is now available for this PR.

- `/ci run` starts upstream CI; `/amd-ci run` starts AMD CI only.
- `/ci retry` retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
- `/amd-ci retry` retries failed jobs in AMD CI for the current PR head. Use `/amd-ci run` when the current head has no AMD CI build.
- `/ci cancel` cancels scheduled or running CI builds for this PR branch; `/amd-ci cancel` does the same for AMD CI only.

<!-- vllm-ci-authorized -->

**gau-nernst** (2026-08-27T03:41:36Z):

@gcanlin Can you rebase onto main? The arm64 docker CI should be fixed by a recent commit iirc.

**gau-nernst** (2026-08-27T03:48:59Z):

/ci run

**github-actions** (2026-08-27T03:49:17Z):

✅ Triggered [Buildkite CI #85744](https://buildkite.com/vllm/ci/builds/85744) for commit `a763990c1717`.

