---
title: '57454'
pageLayout: default
slug: thoughts/inference/vllm-prs/57454
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/57454.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#57454](https://github.com/aarnphm/aarnphm.github.io/issues/57454) \[Bugfix\]\[DSv4.1\] Preserve NaN-scored candidate block indices

[vllm-project/vllm#57454](https://github.com/vllm-project/vllm/pull/57454) · WoosukKwon · MERGED · created 2026-09-17T23:08:27Z · merged 2026-09-18T02:00:10Z

DeepSeek-V4.1 TP4 MegaAttention can fail startup with a DeepGEMM sparse-slot assertion during dummy prefill/autotuning. The candidate-source logits contain NaNs. Although `torch.topk` selects their block IDs, `value > -inf` discards them, leaving only the pinned newest block. Expanding that underfilled row repeats its last block inside the prefix DeepGEMM expects to be unique (389 copies of block 388 in the captured failing rows).

Exclude only the `-inf` padding sentinel when storing selected candidates. This retains NaN-scored block IDs, keeps the active prefix unique for the observed configuration, and works with the existing DeepGEMM dependency. Finite-score selection is unchanged; this adds no kernel, buffer, fallback, or host synchronization. It does not sanitize logits or fix the upstream numerical source of dummy-run NaNs.

Extend the existing candidate suite with empty/partial rows and aligned/unaligned starts under zero/NaN scores, and update its reference to retain selected NaN entries.

Validation on GB200:

- Both new NaN regression cases fail against the original selector; both finite control cases pass.
- All 13 candidate tests pass on PR commit `707b81b021`; scoped pre-commit hooks (including mypy) pass.
- Four captured TP-rank inputs replay through unmodified DeepGEMM after the fix, recovering 389 unique blocks in each previously failing row.
- Selector checks at 8K, 50K and 100K cover random, zero, NaN and mixed-NaN scores; finite outputs match the original exactly.
- The identical selector change on the previously validated model worktree passes real-weight TP4 MegaAttention startup with sparse logits, DSpark5, autotuning and FULL target/draft graphs. Short (18-token) and long (16,526-token) chat prompts both return the expected answer `4`. This is a serving smoke, not a full accuracy evaluation or a full-model build of current main. No full accuracy evaluation was repeated.

Duplicate-work check: [#56558](https://github.com/aarnphm/aarnphm.github.io/issues/56558) optimizes the candidate-source selector but explicitly preserves NaN exclusion. [#50201](https://github.com/aarnphm/aarnphm.github.io/issues/50201) and [#49897](https://github.com/aarnphm/aarnphm.github.io/issues/49897) address the downstream token top-k kernel. None fixes this candidate-store predicate. [vllm-project/DeepGEMM#16](https://github.com/vllm-project/DeepGEMM/issues/16) instead adds repeated-index support to the native metadata API; this PR prevents vLLM from constructing the invalid active prefix for the observed NaN case and needs no dependency update. No linked issue.

AI assistance: implementation, investigation and validation used OpenAI Codex. The human submitter confirmed reviewing every changed line of commit `707b81b021` and running the relevant GPU tests.

---

## comments

**WoosukKwon** (2026-09-18T00:09:30Z):

/ci run

**github-actions** (2026-09-18T00:09:48Z):

✅ Triggered [Buildkite CI #89744](https://buildkite.com/vllm/ci/builds/89744) for commit `5d5aadfdb724`.

