---
title: '57906'
pageLayout: default
slug: thoughts/inference/vllm-prs/57906
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/57906.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#57906](https://github.com/aarnphm/aarnphm.github.io/issues/57906) \[Bugfix\]\[ROCm\]\[DSv4.1\] Disable SWA bounded replay on ROCm

[vllm-project/vllm#57906](https://github.com/vllm-project/vllm/pull/57906) · Fangzhou-Ai · MERGED · created 2026-09-21T08:07:25Z · merged 2026-09-21T10:29:15Z

## Purpose

Disable SWA bounded replay on ROCm, where it currently faults the GPU.

SWA bounded replay ([#56227](https://github.com/aarnphm/aarnphm.github.io/issues/56227), on by default) keeps the sliding-window KV out of prefix caching and rebuilds it after a prefix hit by padding the replayed tokens’ slots in the prefix-cacheable groups. Correctness depends on the consuming prefill kernel clamping the window; that clamp landed in the FlashInfer and FlashMLA paths.

The ROCm sparse SWA metadata builders (`DeepseekV4ROCMAiterSparseSWAMetadataBuilder` in both `deepseek_v4/amd/rocm.py` and `deepseek_v41/amd/rocm.py`) only gained the `replay_start` kwarg and forward it to the base builder. Nothing on the ROCm prefill path clamps the window, so the first prefix hit that carries a replay start walks off the padded slots and raises `HSA_STATUS_ERROR_MEMORY_FAULT`, which takes down the worker and then the engine core:



```
Memory access fault by GPU node-6 (Agent handle: 0x5621a810) on address 0x7ba8c13ff000. Reason: Unknown.
RuntimeError: Worker failed with error 'CUDA error: an illegal memory access was encountered'
RuntimeError: Engine core process EngineCore died unexpectedly.
```

This adds a ROCm guard alongside the existing model-runner-V2 and prefill-context-parallel guards in `DeepseekV4Attention`. Prefix caching itself is unaffected — the sliding-window cache simply takes part in it again, which is the [pre-#56227](https://github.com/pre-/aarnphm.github.io/issues/56227) behavior. The guard should be removed once the ROCm prefill kernels clamp the window too.

## Test Plan

DeepSeek-V4.1-Flash on MI355X (gfx950), ROCm 10.0, torch 2.12.0+rocm10.0.0, Triton 3.8.0, TP=4, `--max-model-len 1048576`, `--max-num-batched-tokens 16384`, DSpark speculative decoding, `--moe-backend aiter`.

Single prompt per request at `max_tokens=1`, sweeping input length across the prefill-chunk boundary:



```
vllm serve deepseek-ai/DeepSeek-V4.1-Flash --tensor-parallel-size 4 \
    --language-model-only --tokenizer-mode deepseek_v41 \
    --moe-backend aiter --gpu-memory-utilization 0.9 \
    --max-model-len 1048576 --max-num-batched-tokens 16384 \
    --speculative-config '{"method":"dspark","num_speculative_tokens":5,...}'
```

## Test Result

Baseline (`swa_bounded_replay` on, the current default):

| Input tokens | Result                                            |
| ------------ | ------------------------------------------------- |
| 4,096        | OK                                                |
| 32,768       | `HSA_STATUS_ERROR_MEMORY_FAULT`, engine core dies |

A prompt that fits in one prefill chunk passes; the fault appears once the prompt spans more than one chunk. Every fault immediately follows the first launch of `_pad_replayed_slots_kernel`. Setting `AMD_SERIALIZE_KERNEL=3` makes the same 32,768-token request pass, which is consistent with reading past the padded slots rather than with a bad constant. `TRITON_HIP_USE_ASYNC_COPY=0` only moves the threshold (32,768 passes, 131,072 still faults), so this is not a Triton async-copy issue.

With this change, on the same build and the same four GPUs:

| Input tokens | Result     |
| ------------ | ---------- |
| 32,768       | OK (1.4s)  |
| 131,072      | OK (5.5s)  |
| 262,144      | OK (10.4s) |
| 314,510      | OK (4.3s)  |

Zero memory faults, and `_pad_replayed_slots_kernel` is never launched.

End to end, a 16-configuration MI355X AgentX sweep (TP=2 and TP=4, concurrency 1-128) failed every point before this change and runs clean after it, with the accuracy arm reporting gsm8k `em_strict` 0.9750 / `em_flexible` 0.9742 over 1319 effective samples.

No new test is added: the two sibling guards in this function (model runner V2, prefill context parallelism) are likewise untested, and exercising this one requires a ROCm device with the model loaded.

---

## comments

**Fangzhou-Ai** (2026-09-21T08:10:36Z):

/ci run

**github-actions** (2026-09-21T08:10:59Z):

✅ @Fangzhou-Ai, CI is now available for this PR.

- `/ci run` starts upstream CI; `/amd-ci run` starts AMD CI only.
- Your branch must contain every commit currently on its upstream target branch. Merge or rebase onto the latest target branch, then rerun the command. Append `--allow-stale` to a run command to test an outdated branch at your own risk.
- `/ci retry` retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
- `/amd-ci retry` retries failed jobs in AMD CI for the current PR head. Use `/amd-ci run` when the current head has no AMD CI build.
- `/ci cancel` cancels scheduled or running CI builds for this PR branch; `/amd-ci cancel` does the same for AMD CI only.

<!-- vllm-ci-authorized -->

**github-actions** (2026-09-21T08:11:07Z):

❌ This PR is 1 commit behind upstream `main`. Your branch must contain every commit currently on upstream `main`. No new CI build was started. Merge or rebase onto the latest `main`, then rerun `/ci run`. To test this branch at your own risk, use `/ci run --allow-stale`.

<!-- vllm-ci-command:5757378791 -->

**Fangzhou-Ai** (2026-09-21T08:11:31Z):

/ci run

**github-actions** (2026-09-21T08:11:51Z):

✅ Triggered [Buildkite CI #90177](https://buildkite.com/vllm/ci/builds/90177) for commit `a054b3809686`.

**shen-shanshan** (2026-09-21T10:08:03Z):

/ci retry

**github-actions** (2026-09-21T10:08:22Z):

✅ Queued 1 failed job(s) for retry in [Buildkite CI #90177](https://buildkite.com/vllm/ci/builds/90177).

