---
title: '59197'
pageLayout: default
slug: thoughts/inference/vllm-prs/59197
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/59197.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#59197](https://github.com/aarnphm/aarnphm.github.io/issues/59197) \[Bugfix\]\[DSv4.1\] Skip SWA bounded replay for KV loads that carry the window

[vllm-project/vllm#59197](https://github.com/vllm-project/vllm/pull/59197) · ivanium · MERGED · created 2026-09-29T09:53:46Z · merged 2026-10-07T22:28:43Z

Also credit to @wangyicong52 who originally reported the issue and helped test the PR.

## Purpose

SWA bounded replay ([#56227](https://github.com/aarnphm/aarnphm.github.io/issues/56227)) replays the last window of every prefix hit, including KV loaded by P/D connectors, which transfer the request’s own sliding-window KV. P/D decode on DeepSeek-V4.1 therefore recomputed the last 128 prompt tokens of every request. The whole-block rounding that replay applies to external hits also shrinks decode’s allocation below what prefill exports, so NIXL’s end-aligned pairing puts prefill block k + 1 into decode block k for the MLA groups whenever N % 128 != 0 (found with a CPU probe of the real scheduler and NIXL code).

## Changes

- The scheduler takes a load that restores every replayed group (`get_loaded_kv_cache_group_ids`, added by [#59504](https://github.com/aarnphm/aarnphm.github.io/issues/59504)) as is: no replay, no whole-block rounding, and no dropping of hits no longer than the window. Async loads record the decision at admission and apply it when the KV arrives; failed loads and store / offloading loads still replay.

- Without the replay, decode relies on the transferred window as laid out, so NIXL and Mooncake now back off the last prompt token for models with bounded replay, as they already do for Mamba: prefill stops at N - 1, decode loads N - 1 tokens and computes the last one. Loading all N tokens and rewinding one after laying the window out for N dropped the block holding position N - W when N % block_size == block_size - 1, and the recomputed token attended to the null block, which DeepSeek-V4.1 does not keep zero.

- The prompt cut keeps two prompt-indexed features working (they were already broken for Mamba P/D): a request with prompt logprobs (e.g. completions with echo and logprobs) is not cut, since it skips reading the prefix cache and the decoder never loads its KV; before, the prefiller returned prompt logprobs for one token fewer than the prompt and the frontend failed. The routed-experts (R3) start is clamped to the prompt length; a start equal to the original length lay past the cut prompt and made the scheduler assert.

This PR stands alone. [#59330](https://github.com/aarnphm/aarnphm.github.io/issues/59330) is the general fix: it applies the backoff to every model (a sliding window has the same rewind problem everywhere, it is only masked by a zero null block) and handles the prompt shapes that only Mamba used to reach (mixed token-id / embeddings prompts, a cut inside a multimodal item, push and Mooncake zero-token loads). DeepSeek-V4.1 is text-only and needs none of those.

## Not a duplicate

[#59142](https://github.com/aarnphm/aarnphm.github.io/issues/59142) (draft, opt-in, Mooncake only) moves replay to prefill and certifies checkpoints; this PR drops the replay of KV that P/D already transfers, for any connector that declares it, with no new config. [#58504](https://github.com/aarnphm/aarnphm.github.io/issues/58504) keeps partial external hits exact by retaining the boundary block; loads carrying the window skip the rounding here.

## Test Plan

- `pytest tests/v1/core/test_prefix_replay.py tests/v1/core/test_scheduler.py tests/v1/kv_connector/unit/{test_nixl_connector_hma,test_nixl_connector,test_nixl_push_connector,test_mooncake_connector,test_mooncake_connector_hma,test_mooncake_connector_hybrid_mamba,test_multi_connector,test_remote_prefill_lifecycle,test_invalid_blocks_correctness,test_kv_load_failure_recovery}.py`: `test_kv_load_carrying_window_does_not_replay` (sync/async, window-long and unaligned hits), the NIXL backoff tests extended to bounded replay and prompt-logprob requests, and the aux output connector test extended to a start past the cut prompt.
- `pre-commit`
- DeepSeek-V4.1-Flash P/D (NIXL, 1P1D TP4 + TP4), two-pass GSM8K 5-shot, concurrency 512; pass 2 re-sends the prompts so decode gets local prefix hits.

## Test Result

- Unit tests: 754 passed.
- GSM8K flexible-extract. “N%32==31” counts pass-2 errors on prompts with N % 32 == 31 (the rewind case); “t < 16” pass-2 errors elsewhere with (N - 1) % 128 < 16, where the kept window lies almost entirely below a decode-side prefix hit (the zeroing race [#59504](https://github.com/aarnphm/aarnphm.github.io/issues/59504) fixed). Main and the earlier rows were run on main [`4e0a414`](https://github.com/aarnphm/aarnphm.github.io/commit/4e0a414c44), this PR on main [`84bcbc6`](https://github.com/aarnphm/aarnphm.github.io/commit/84bcbc6264):

| Build                                                                                                        | Pass 1          | Pass 2          | N%32==31 | t < 16  |
| ------------------------------------------------------------------------------------------------------------ | --------------- | --------------- | -------- | ------- |
| main                                                                                                         | 0.9174 / 0.9249 | 0.9219 / 0.9189 | 5 / 6    | 18 / 15 |
| replay skip only, without the backoff or [#59504](https://github.com/aarnphm/aarnphm.github.io/issues/59504) | 0.9181          | 0.9052          | 13       | 23      |
| this PR                                                                                                      | 0.9212          | 0.9227          | 5        | 13      |

AI assistance (Claude Code) was used for this PR.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---

## comments

**wangyicong52** (2026-09-29T13:34:46Z):

Hi @ivanium, thanks for working on this PR!

We ran an H20 P/D benchmark comparing the following configurations:

- **baseline**: nightly `a9eafde59c` with merged [#58132](https://github.com/aarnphm/aarnphm.github.io/issues/58132).
- **This PR**: the same baseline plus [#59197](https://github.com/aarnphm/aarnphm.github.io/issues/59197) at `3b4865e2c1`.
- During testing, a release-only Mooncake pull could incorrectly report receive completion even though no local blocks were loaded. We applied a small fix to suppress that completion.

### Environment

- Model: `DeepSeek-V4.1-Flash`
- Image: `vllm/vllm-openai:nightly-a9eafde59cbd55182dc2265cc398b7186f0a0eaa`
- Hardware: Prefill 8x NVIDIA H20 with TP8; Decode 8x NVIDIA H20 with TP1 + DP8 + EP8

### vLLM startup parameters

Prefill:



```bash
vllm serve /data00/models/DeepSeek-V4.1-Flash \
  --host 0.0.0.0 \
  --port 8001 \
  --served-model-name deepseek-v4.1-flash \
  --trust-remote-code \
  --tokenizer-mode deepseek_v41 \
  --engram-config '{"cpu_offload":true}' \
  --max-model-len auto \
  --enable-cumem-allocator \
  --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_producer","kv_load_failure_policy":"fail","kv_buffer_device":"cuda","kv_connector_extra_config":{"mooncake_protocol":"rdma","device_name":"<rdma-device-list>","enforce_handshake_compat":false}}' \
  --block-size 64 \
  --swa-bounded-replay \
  --enable-prefix-caching \
  --tool-call-parser deepseek_v41 \
  --enable-auto-tool-choice \
  --reasoning-parser deepseek_v41 \
  --mm-encoder-tp-mode data \
  --api-server-count 1 \
  --tensor-parallel-size 8 \
  --max-num-batched-tokens 8192 \
  --max-num-seqs 256 \
  --gpu-memory-utilization 0.92 \
  --no-async-scheduling \
  --compilation-config '{"cudagraph_mode":"NONE"}' \
  --speculative-config '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"block","enable_adaptive_verification":false}'
```

Decode:



```bash
vllm serve /data00/models/DeepSeek-V4.1-Flash \
  --host 0.0.0.0 \
  --port 8002 \
  --served-model-name deepseek-v4.1-flash \
  --trust-remote-code \
  --tokenizer-mode deepseek_v41 \
  --engram-config '{"cpu_offload":true}' \
  --max-model-len auto \
  --enable-cumem-allocator \
  --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_consumer","kv_load_failure_policy":"fail","kv_buffer_device":"cuda","kv_connector_extra_config":{"mooncake_protocol":"rdma","device_name":"<rdma-device-list>","enforce_handshake_compat":false}}' \
  --block-size 64 \
  --swa-bounded-replay \
  --enable-prefix-caching \
  --tool-call-parser deepseek_v41 \
  --enable-auto-tool-choice \
  --reasoning-parser deepseek_v41 \
  --mm-encoder-tp-mode data \
  --api-server-count 1 \
  --tensor-parallel-size 1 \
  --data-parallel-size 8 \
  --pipeline-parallel-size 1 \
  --kv-cache-dtype fp8_ds_mla \
  --cp-kv-cache-interleave-size 256 \
  --enable-expert-parallel \
  --max-num-batched-tokens 2048 \
  --max-num-seqs 16 \
  --gpu-memory-utilization 0.92 \
  --no-disable-hybrid-kv-cache-manager \
  --speculative-config '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"block","enable_adaptive_verification":true}' \
  --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \
  --enable-per-request-metrics \
  --per-request-spec-decode-metrics summary
```

### Benchmark templates

Both prefix caches were reset before warmup and again before each measured tier. Prefill and E2E covered exact 4K, 8K, 16K, and 32K input lengths. Decode additionally covered 1K; the 4K and 8K Decode tiers were repeated, and the tables below use the arithmetic mean of both runs. Prefill warmup used 8 requests; Decode and E2E warmup used 16 requests.

Prefill:



```bash
vllm bench serve \
  --backend vllm \
  --base-url <router-endpoint> \
  --model /data00/models/DeepSeek-V4.1-Flash \
  --served-model-name deepseek-v4.1-flash \
  --tokenizer /data00/models/DeepSeek-V4.1-Flash \
  --tokenizer-mode deepseek_v41 \
  --trust-remote-code \
  --dataset-name random \
  --random-input-len <4096|8192|16384|32768> \
  --random-output-len 1 \
  --random-range-ratio 0 \
  --num-prompts 32 \
  --request-rate 0.2 \
  --max-concurrency 4 \
  --seed <1578720+input_len> \
  --temperature 0 \
  --ignore-eos \
  --disable-tqdm \
  --save-result \
  --save-detailed
```

Decode:



```bash
vllm bench serve \
  --backend vllm \
  --base-url <router-endpoint> \
  --model /data00/models/DeepSeek-V4.1-Flash \
  --served-model-name deepseek-v4.1-flash \
  --tokenizer /data00/models/DeepSeek-V4.1-Flash \
  --tokenizer-mode deepseek_v41 \
  --trust-remote-code \
  --dataset-name prefix_repetition \
  --prefix-repetition-prefix-len <input_len-100> \
  --prefix-repetition-suffix-len 100 \
  --prefix-repetition-num-prefixes 1 \
  --prefix-repetition-output-len 1024 \
  --num-prompts 80 \
  --request-rate inf \
  --max-concurrency 16 \
  --seed 0 \
  --temperature 0 \
  --ignore-eos \
  --disable-tqdm \
  --save-result \
  --save-detailed
```

Random end to end:



```bash
vllm bench serve \
  --backend vllm \
  --base-url <router-endpoint> \
  --model /data00/models/DeepSeek-V4.1-Flash \
  --served-model-name deepseek-v4.1-flash \
  --tokenizer /data00/models/DeepSeek-V4.1-Flash \
  --tokenizer-mode deepseek_v41 \
  --trust-remote-code \
  --dataset-name random \
  --random-input-len <4096|8192|16384|32768> \
  --random-output-len 512 \
  --random-range-ratio 0 \
  --num-prompts 80 \
  --request-rate inf \
  --max-concurrency 16 \
  --seed 0 \
  --temperature 0 \
  --ignore-eos \
  --extra-body '{"seed":0}' \
  --disable-tqdm \
  --save-result \
  --save-detailed
```

### Results

All deltas below are **This PR versus baseline**. Negative latency is better; positive throughput is better.

#### Prefill

| Input | Baseline Mean TTFT | This PR Mean TTFT |  Delta | Baseline P99 | This PR P99 |  Delta |
| ----: | -----------------: | ----------------: | -----: | -----------: | ----------: | -----: |
|    4K |         1023.53 ms |         983.02 ms | -3.96% |   1336.48 ms |  1285.07 ms | -3.85% |
|    8K |         1534.89 ms |        1484.04 ms | -3.31% |   2037.42 ms |  2041.76 ms | +0.21% |
|   16K |         2261.53 ms |        2219.55 ms | -1.86% |   3400.94 ms |  3355.77 ms | -1.33% |
|   32K |         4385.75 ms |        4364.25 ms | -0.49% |   7276.51 ms |  7240.80 ms | -0.49% |

#### Decode

| Input | Runs | Baseline TPS | This PR TPS |   Delta | Baseline Mean TTFT | This PR Mean TTFT |   Delta |
| ----: | ---: | -----------: | ----------: | ------: | -----------------: | ----------------: | ------: |
|    1K |    1 |      1600.47 |     1496.51 |  -6.50% |          810.40 ms |         822.80 ms |  +1.53% |
|    4K |    2 |      1666.46 |     1692.54 |  +1.57% |         1096.91 ms |         912.71 ms | -16.79% |
|    8K |    2 |      2474.52 |     2202.69 | -10.99% |         1421.68 ms |        1138.19 ms | -19.94% |
|   16K |    1 |      2255.56 |     2401.11 |  +6.45% |         1298.53 ms |        1369.72 ms |  +5.48% |
|   32K |    1 |      2084.48 |     2199.01 |  +5.49% |         1904.61 ms |        1912.57 ms |  +0.42% |

| Input | Baseline Mean/P99 TPOT | This PR Mean/P99 TPOT |    Mean/P99 delta | Baseline acceptance | This PR acceptance |    Delta |
| ----: | ---------------------: | --------------------: | ----------------: | ------------------: | -----------------: | -------: |
|    1K |      7.573 / 18.489 ms |     7.983 / 21.727 ms |  +5.42% / +17.52% |              76.81% |             71.54% | -5.27 pp |
|    4K |      7.424 / 14.717 ms |     7.255 / 15.325 ms |   -2.29% / +4.14% |              87.52% |             83.53% | -3.98 pp |
|    8K |       5.057 / 5.495 ms |      5.887 / 8.286 ms | +16.40% / +50.80% |              94.75% |             92.87% | -1.88 pp |
|   16K |       5.473 / 6.710 ms |      5.257 / 6.164 ms |   -3.96% / -8.14% |              93.62% |             93.88% | +0.26 pp |
|   32K |       5.447 / 6.913 ms |      5.362 / 6.122 ms |  -1.55% / -11.44% |              93.73% |             94.11% | +0.38 pp |

Across 4K/8K/16K/32K, Output TPS is effectively unchanged at 2120.26 versus 2123.84 (+0.17%). Mean/P99 TPOT changes from 5.850/8.459 ms to 5.940/8.974 ms (+1.53%/+6.10%), while acceptance changes from 92.40% to 91.10% (-1.31 percentage points). The 4K and 8K entries use the arithmetic mean of two runs.

#### Random E2E, OSL 512

| Input | Baseline TPS | This PR TPS |   Delta | Baseline Mean TTFT | This PR Mean TTFT |   Delta | Baseline Mean TPOT | This PR Mean TPOT |   Delta |
| ----: | -----------: | ----------: | ------: | -----------------: | ----------------: | ------: | -----------------: | ----------------: | ------: |
|    4K |      1078.03 |     1196.88 | +11.02% |         1671.88 ms |        1875.51 ms | +12.18% |           9.759 ms |          8.602 ms | -11.86% |
|    8K |       811.48 |      819.63 |  +1.00% |         5673.00 ms |        5962.49 ms |  +5.10% |           7.289 ms |          6.533 ms | -10.38% |
|   16K |       432.26 |      435.14 |  +0.66% |        14842.71 ms |       14985.39 ms |  +0.96% |           4.985 ms |          4.687 ms |  -5.97% |
|   32K |       218.54 |      218.60 |  +0.03% |        32026.07 ms |       32091.27 ms |  +0.20% |           4.300 ms |          4.171 ms |  -3.00% |

### Full-hit SWA boundary follow-up

With block size 64 and SWA window 128, we also tested exact token lengths around the suspected full-hit boundary:

| Prompt tokens |       Baseline |        This PR |
| ------------: | -------------: | -------------: |
|          1022 |     2/2 passed |     2/2 passed |
|      **1023** | **2/2 passed** | **0/2 passed** |
|          1024 |     2/2 passed |     2/2 passed |
|          1025 |     2/2 passed |     2/2 passed |
|          4094 |     2/2 passed |     2/2 passed |
|      **4095** | **2/2 passed** | **0/2 passed** |
|          4096 |     2/2 passed |     2/2 passed |
|          4097 |     2/2 passed |     2/2 passed |
|      **8191** | **2/2 passed** | **0/2 passed** |

At 1023/4095/8191 (`prompt_tokens % block_size == block_size - 1`), Prefill succeeded but Decode returned HTTP 400 because the logits contained NaN. After a full KV load, this PR skips replay but still rewinds one token for sampling; the resumed query then needs the token at `N - 128`, whose SWA block has already been retired.

The smallest safe fix appears to be retaining at least one additional historical token for replay-enabled SWA specs on both Prefill and Decode. Decode-only retention cannot recover a block already discarded by Prefill. This boundary issue does not directly explain the aligned 4096/8192 performance results.

These preliminary results are for reference only. Across the tested input lengths, Decode throughput is effectively neutral, with lower Mean TTFT but slightly higher TPOT and lower acceptance overall. GSM8K accuracy testing has not been run.

**ZJY0516** (2026-10-01T14:04:12Z):

/ci run —allow-stale

**github-actions** (2026-10-01T14:04:56Z):

✅ Triggered [Buildkite CI #92415](https://buildkite.com/vllm/ci/builds/92415) for commit `d8f4df6483b5`.

⚠️ This PR is 134 commits behind upstream `main`. Running CI at your own risk because `--allow-stale` was requested; outdated CI configuration may cause failures. Before merging, merge or rebase onto the latest `main`, then rerun `/ci run` on the latest PR commit.

**njhill** (2026-10-01T22:01:10Z):

/ci run

**github-actions** (2026-10-01T22:01:33Z):

✅ Triggered [Buildkite CI #92499](https://buildkite.com/vllm/ci/builds/92499) for commit `35c9144e4fdb`.

**mergify** (2026-10-02T03:49:46Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @ivanium.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**ivanium** (2026-10-06T06:42:32Z):

/ci run

**github-actions** (2026-10-06T06:42:51Z):

✅ Triggered [Buildkite CI #93043](https://buildkite.com/vllm/ci/builds/93043) for commit `ab2e2230a165`.

**vllm-agent** (2026-10-06T06:54:51Z):

<!-- ci-selector-shadow -->

### CI selector (shadow): 133 test steps (172 jobs) instead of 83 (100 jobs)

Shadow mode: this changes nothing about what CI runs. It shows what the evidence-based selector would pick for this PR, next to today’s rules. [How it works](https://github.com/vllm-project/ci-infra/tree/main/buildkite/ci_selector).

**Feedback welcome:** reply here if it would skip a step this change needs, or runs something unrelated.

| steps (jobs)           | Today’s rules | Selector  | Would skip | Would add |
| ---------------------- | ------------- | --------- | ---------- | --------- |
| NVIDIA, CPU and others | 83 (100)      | 133 (172) | 13 (21)    | 63 (93)   |
| AMD mirrors            | 85 (96)       | 122 (167) | 13 (16)    | 50 (87)   |

<details><summary>Selector would run (133)</summary>

- `amd-kernels-mi355`
- `amd-lm-eval-small-models-harness`
- `amd-qwen3-next-mtp-async-eplb-accuracy`
- `async-engine-inputs-utils-worker`
- `basic-correctness` ×2
- `basic-correctness-cpu-offload`
- `basic-correctness-cumem`
- `basic-correctness-prefetch-offload`
- `basic-correctness-sleep-mode`
- `basic-models-tests-extra-initialization` ×14
- `basic-models-tests-initialization`
- `basic-models-tests-other`
- `batch-invariance-b200`
- `batch-invariance-h100`
- `benchmarks-cli-test`
- `cpu-multimodal-config`
- `crosslayer-kv-layout-distributed-nixlconnector-pd-accuracy-tests-4-gpus`
- `cudagraph`
- `distributed-compile-comm-4-gpus`
- `distributed-compile-rpc-tests-2-gpus`
- `distributed-dp-tests-2-gpus`
- `distributed-dp-tests-4-gpus`
- `distributed-flashinfer-nixlconnector-pd-accuracy-4-gpus`
- `distributed-model-tests-2-gpus` ×3
- `distributed-mooncakeconnector-pd-accuracy-4-gpus`
- `distributed-nixlconnector-pd-accuracy-4-gpus`
- `distributed-tests-8xh100`
- `distributed-torchrun-examples-4-gpus`
- `distributed-torchrun-shutdown-tests-2-gpus`
- `dp-ep-distributed-nixlconnector-pd-accuracy-tests-4-gpus`
- `e2e-core-1-gpu`
- `e2e-core-large-memory`
- `e2e-scheduling-1-gpu`
- `e2e-scheduling-accuracy-1-gpu`
- `elastic-ep-scaling-test`
- `engine`
- `engine-1-gpu`
- `entrypoints-integration-api-server` ×4
- `entrypoints-integration-api-server-generate`
- `entrypoints-integration-api-server-openai-chat_completion`
- `entrypoints-integration-api-server-openai-completion`
- `entrypoints-integration-llm`
- `entrypoints-integration-multimodal`
- `entrypoints-integration-pooling`
- `entrypoints-integration-responses-api`
- `entrypoints-integration-speech_to_text`
- `entrypoints-unit-tests`
- `examples`
- `extract-hidden-states-integration`
- `extract-hidden-states-integration-2-gpus`
- `fault-tolerance-e2e-2xh100`
- `fusion-e2e-config-sweep-h100`
- `fusion-e2e-quick-h100`
- `fusion-e2e-tp2-ar-rms-config-sweep-h100`
- `fusion-e2e-tp2-ar-rms-dynamo-partition-amd`
- `fusion-e2e-tp2-ar-rms-inductor-partition-amd`
- `fusion-e2e-tp2-asynctp-config-sweep-h100`
- `fusion-e2e-tp2-b200`
- `fusion-e2e-tp2-quick-h100`
- `hybrid-ssm-nixlconnector-pd-accuracy-tests-4-gpus`
- `hybrid-ssm-nixlconnector-pd-prefix-cache-2-gpus`
- `kernels-b200` ×3
- `kernels-deepgemm-test-h100`
- `kernels-root-misc-test-b200`
- `kimi-k3-prefix-cache-4xb200` ×2
- `kv-offload-large`
- `kv-offload-medium`
- `kv-offload-small`
- `language-models-tests-extra-standard` ×2
- `language-models-tests-granite-l4-compatibility`
- `language-models-tests-hybrid` ×2
- `language-models-tests-standard`
- `lm-eval-dspark-watermark-2xh100`
- `lm-eval-small-models`
- `lm-eval-turboquant-k3v4nc`
- `lm-eval-turboquant-k8v4`
- `lm-eval-turboquant-t3nc`
- `lm-eval-turboquant-t4nc`
- `lm-eval-watermarking`
- `lora` ×4
- `lora-tp-distributed` ×4
- `metrics-tracing-2-gpus`
- `model-executor`
- `mooncake-ec-tcp-e2e-2-gpus`
- `mrcr-eval-small-models`
- `multi-modal-accuracy-eval-small-models`
- `multi-modal-models-standard-1-qwen2`
- `multi-modal-models-standard-2-qwen3-gemma`
- `multi-modal-models-standard-3-llava-qwen2-vl`
- `multi-modal-models-standard-4-other-whisper`
- `multiconnector-nixl-offloading-pd-accuracy-2-gpus`
- `multiconnector-nixl-offloading-pd-edge-cases-2-gpus`
- `nixlconnector-pd-edge-cases-2-gpus`
- `openai-api-correctness`
- `pipeline-context-parallelism-4-gpus`
- `plugin-tests-2-gpus`
- `push-nixlconnector-pp-prefill-pd-accuracy-4-gpus`
- `pytorch-compilation-dynamic-shapes`
- `pytorch-compilation-unit-tests`
- `pytorch-compilation-unit-tests-h100`
- `pytorch-fullgraph-cudagraph-l4-compatibility`
- `pytorch-fullgraph-test`
- `quantization` ×4
- `quantized-models-test`
- `quantized-moe-test-b200`
- `rayexecutorv2-4-gpus`
- `regression`
- `replayssm-e2e`
- `rust-frontend-core-correctness`
- `rust-frontend-distributed`
- `rust-frontend-openai-coverage`
- `rust-frontend-serve-admin-coverage`
- `rust-frontend-tool-use`
- `samplers-multimodal-beam-search`
- `samplers-test`
- `scale-out-ec-e2e-2-gpus`
- `sharded-rdt-weight-transfer`
- `spec-decode-draft-model` ×4
- `spec-decode-eagle-1-deepseek-qwen`
- `spec-decode-eagle-2-llama3-qwen-vl-other`
- `spec-decode-mtp-deepseek-mimo`
- `spec-decode-mtp-gemma4`
- `spec-decode-mtp-qwen3-5`
- `spec-decode-ngram-suffix`
- `spec-decode-speculators`
- `v1-core`
- `v1-executor-worker`
- `v1-kv-connectors` ×4
- `v1-logits-oracle`
- `v1-metrics-lmeval`
- `v1-others-cpu`
- `v1-sample`
- `v1-spec-decode`

</details>

<details><summary>Would skip (today's rules run them) (13)</summary>

- `ascend-npu-test`
- `basic-models-test-other-cpu`
- `cpu-language-generation-and-pooling-model-tests` ×3
- `cpu-params-env-tokenizers-parser`
- `cpu-reasoning-renderers`
- `cpu-tool-parsers`
- `distributed-comm-ops`
- `kernels-fla-ops-test-b200`
- `kernels-mhc-test-b200`
- `multi-modal-processor` ×4
- `multi-modal-processor-cpu` ×4
- `pytorch-compilation-passes-unit-tests`
- `v1-kv-offload`

</details>

<details><summary>Would add (today's rules do not run them) (63)</summary>

- `amd-kernels-mi355` (Python record)
- `amd-lm-eval-small-models-harness` (Python record)
- `amd-qwen3-next-mtp-async-eplb-accuracy` (Python record)
- `basic-models-tests-extra-initialization` ×14 (Python record)
- `batch-invariance-b200` (Python record)
- `batch-invariance-h100` (Python record)
- `cudagraph` (Python record)
- `distributed-model-tests-2-gpus` ×3 (Python record)
- `engine` (code map)
- `engine-1-gpu` (Python record)
- `entrypoints-unit-tests` (Python record)
- `examples` (Python record)
- `extract-hidden-states-integration` (Python record)
- `extract-hidden-states-integration-2-gpus` (Python record)
- `fault-tolerance-e2e-2xh100` (Python record)
- `fusion-e2e-config-sweep-h100` (Python record)
- `fusion-e2e-quick-h100` (Python record)
- `fusion-e2e-tp2-ar-rms-config-sweep-h100` (Python record)
- `fusion-e2e-tp2-ar-rms-dynamo-partition-amd` (Python record)
- `fusion-e2e-tp2-ar-rms-inductor-partition-amd` (Python record)
- `fusion-e2e-tp2-asynctp-config-sweep-h100` (Python record)
- `fusion-e2e-tp2-b200` (Python record)
- `fusion-e2e-tp2-quick-h100` (Python record)
- `kernels-b200` ×3 (Python record)
- `kernels-deepgemm-test-h100` (Python record)
- `kv-offload-large` (Python record)
- `kv-offload-medium` (Python record)
- `kv-offload-small` (Python record)
- `language-models-tests-extra-standard` ×2 (Python record)
- `lm-eval-dspark-watermark-2xh100` (Python record)
- `lm-eval-small-models` (Python record)
- `lm-eval-turboquant-k3v4nc` (Python record)
- `lm-eval-turboquant-k8v4` (Python record)
- `lm-eval-turboquant-t3nc` (Python record)
- `lm-eval-turboquant-t4nc` (Python record)
- `lm-eval-watermarking` (Python record)
- `lora` ×4 (Python record)
- `lora-tp-distributed` ×4 (Python record)
- `model-executor` (Python record)
- `mrcr-eval-small-models` (Python record)
- `openai-api-correctness` (Python record)
- `plugin-tests-2-gpus` (code map)
- `quantization` ×4 (Python record)
- `quantized-models-test` (Python record)
- `quantized-moe-test-b200` (Python record)
- `rayexecutorv2-4-gpus` (Python record)
- `replayssm-e2e` (Python record)
- `rust-frontend-core-correctness` (Python record)
- `rust-frontend-openai-coverage` (Python record)
- `rust-frontend-serve-admin-coverage` (Python record)
- `rust-frontend-tool-use` (Python record)
- `samplers-multimodal-beam-search` (Python record)
- `samplers-test` (Python record)
- `scale-out-ec-e2e-2-gpus` (Python record)
- `sharded-rdt-weight-transfer` (Python record)
- `spec-decode-draft-model` ×4 (Python record)
- `spec-decode-eagle-1-deepseek-qwen` (Python record)
- `spec-decode-eagle-2-llama3-qwen-vl-other` (Python record)
- `spec-decode-mtp-deepseek-mimo` (Python record)
- `spec-decode-mtp-gemma4` (Python record)
- `spec-decode-mtp-qwen3-5` (Python record)
- `spec-decode-ngram-suffix` (Python record)
- `spec-decode-speculators` (Python record)

</details>

<details><summary>AMD mirrors: would skip (13)</summary>

- `distributed-comm-ops`
- `distributed-compile-comm-4-gpus`
- `fault-tolerance-e2e-2xh100`
- `gemm-rs-ar-2xb200`
- `kernels-fla-ops-test-b200`
- `kernels-fusedmoe-layer-test-2-b200s`
- `kernels-fusedmoe-layer-test-2-h100s`
- `kernels-mhc-test-b200`
- `kernels-minimax-reduce-rms-test-2-gpus`
- `multi-modal-processor` ×4
- `platform-tests`
- `pytorch-compilation-passes-unit-tests`
- `v1-kv-offload`

</details>

<details><summary>AMD mirrors: would add (50)</summary>

- `basic-models-tests-extra-initialization` ×14 (Python record)
- `cudagraph` (Python record)
- `distributed-model-tests-2-gpus` ×3 (Python record)
- `engine` (code map)
- `engine-1-gpu` (Python record)
- `entrypoints-unit-tests` (Python record)
- `examples` (Python record)
- `extract-hidden-states-integration` (Python record)
- `fusion-e2e-config-sweep-h100` (Python record)
- `fusion-e2e-quick-h100` (Python record)
- `jit-monitor-no-runtime-jit` (Python record)
- `kernels-moe-test` ×5 (Python record)
- `kernels-quantization-test` ×6 (Python record)
- `kv-offload-large` (Python record)
- `kv-offload-medium` (Python record)
- `kv-offload-small` (Python record)
- `language-models-tests-extra-standard` ×2 (Python record)
- `lm-eval-dspark-watermark-2xh100` (Python record)
- `lm-eval-small-models` (Python record)
- `lm-eval-turboquant-k3v4nc` (Python record)
- `lm-eval-turboquant-k8v4` (Python record)
- `lm-eval-turboquant-t3nc` (Python record)
- `lm-eval-turboquant-t4nc` (Python record)
- `lm-eval-watermarking` (Python record)
- `lora` ×4 (Python record)
- `lora-tp-distributed` ×4 (Python record)
- `model-executor` (Python record)
- `mrcr-eval-small-models` (Python record)
- `openai-api-correctness` (Python record)
- `plugin-tests-2-gpus` (code map)
- `quantization` ×4 (Python record)
- `quantized-models-test` (Python record)
- `quantized-moe-test-b200` (Python record)
- `rayexecutorv2-4-gpus` (Python record)
- `rust-frontend-core-correctness` (Python record)
- `rust-frontend-openai-coverage` (Python record)
- `rust-frontend-serve-admin-coverage` (Python record)
- `rust-frontend-tool-use` (Python record)
- `samplers-multimodal-beam-search` (Python record)
- `samplers-test` (Python record)
- `scale-out-ec-e2e-2-gpus` (Python record)
- `sharded-rdt-weight-transfer` (Python record)
- `spec-decode-draft-model` ×4 (Python record)
- `spec-decode-eagle-1-deepseek-qwen` (Python record)
- `spec-decode-eagle-2-llama3-qwen-vl-other` (Python record)
- `spec-decode-mtp-deepseek-mimo` (Python record)
- `spec-decode-mtp-gemma4` (Python record)
- `spec-decode-mtp-qwen3-5` (Python record)
- `spec-decode-ngram-suffix` (Python record)
- `spec-decode-speculators` (Python record)

</details>

<sub>9 changed files · base `6ad2e06c4a` · head `ab2e2230a1` · Python record: build 92956 at `18c8a65edd` · kernel record: table [`18c8a65`](https://github.com/aarnphm/aarnphm.github.io/commit/18c8a65edd) (build 92956), map [`18c8a65`](https://github.com/aarnphm/aarnphm.github.io/commit/18c8a65edd) · not counted: 10 build steps, 5 A100 steps the generator no longer emits, 122 optional steps the selector would also run</sub>

