---
title: '57652'
pageLayout: default
slug: thoughts/inference/vllm-prs/57652
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/57652.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#57652](https://github.com/aarnphm/aarnphm.github.io/issues/57652) \[KV-Offloading\]\[TP\] : Expand replicated_layout detection to multi-group MLA

[vllm-project/vllm#57652](https://github.com/vllm-project/vllm/pull/57652) · varun-sundar-rabindranath · MERGED · created 2026-09-19T03:25:28Z · merged 2026-09-30T13:11:04Z

## Purpose

KV offloading for MLA models + TP need only offload KV values from one TP rank.
This is controlled by replicated_layout determination in `vllm/distributed/kv_transfer/kv_connector/v1/offloading/config.py`

Building on top of @almogtavor ‘s work [vllm-project/vllm#50667](https://github.com/vllm-project/vllm/pull/50667) (which was eventually closed) , this PR expand the replicated_layout check to include,

- SlidingWindowMLASpec
- Multi-group MLASpec / SlidingWindowMLASpec
  models.

Notably,

- deepseek-ai/DeepSeek-V4-Flash
- deepseek-ai/DeepSeek-V3.2-Exp and
- zai-org/GLM-5.3
  fall in this category.

### Changes:

- `AttentionSpec.max_tp_shards: int | None = None`: how many distinct shards this cache splits into across TP. A value of means, all TP ranks are replicas. None means unknown and is treated as not replicated.
  - It’s opt-in: no spec class has a default that claims replication, so a layer that forgets it only loses dedup, never correctness. This is the same pattern as MambaSpec.tp_replicated.
  - max_tp_shards=1 is set only where the cache has been checked to be identical on every TP rank:
    - generic MLAAttention (mla_attention.py), which covers the DSv3 / V3.2 / GLM-5.3 latent cache;
    - the DSv3.2 indexer cache (deepseek_v2.py; wk_weights_proj uses disable_tp=True);
    - DeepSeek-V4 attention and indexer caches (deepseek_v4/attention.py);
    - the DeepSeek-V4 compressor state (fused_wkv_wgate uses disable_tp=True);
    - the DeepSeek-V4 SWA cache (sparse_swa.py).
- `KVCacheConfig.kv_tp_replicas: int = 1` : get_kv_cache_configs sets it through `kv_cache_groups_tp_replicas(groups, tp_size, dcp_size)` - The gcd of each layer’s `tp_size // max_tp_shards`, taken per layer through `iter_layer_specs`. A layer counts as 1 if it isn’t an attention spec, has no value set, or is DCP-sharded under DCP.
- Offloading: replicated_layout requires tp > 1 and kv_tp_replicas == tp. T

#### Justification for adding a KVCacheConfig member

MLAAttentionSpec shares `uniform_type_base_spec=FullAttentionSpec`, so one `UniformTypeKVCacheSpecs` group can mix MLA and GQA layers. `generate_scheduler_kv_cache_config` keeps an arbitrary layer spec, so the value has to be computed before that. Otherwise the scheduler could see an MLA spec and wrongly de-duplicate head-sharded KV.

## Test Plan

Add unit tests.

lm-eval tests : eval script
[eval.sh](https://github.com/user-attachments/files/32408750/eval.sh)
Methodology : Run models with `vllm serve` and disk-offloading. Run lm-eval 3 times.
lm-eval run 1 : cold run. The disk is filled.
<clear gpu and cpu KV caches with reset-prefix-cache endpoint>
lm-eval run 2 : warm run. offloaded KV data is loaded from Disk
<clear gpu and cpu KV caches with reset-prefix-cache endpoint>
lm-eval run 3 : warm run. offloadead KV data is loaded from Disk (this is just a confirmation run)

## Test Result

Unit tests pass.

lm-evals:

### DSV2-lite



```
local-completions ({'base_url': 'http://localhost:8000/v1/completions', 'model': 'deepseek-ai/DeepSeek-V2-Lite', 'tokenized_requests': False, 'num_concurrent': 1000}), gen_kwargs: ({'temperature': 0.0}), limit: 500.0, num_fewshot: 25, batch_size: 1
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
|gsm8k|      3|flexible-extract|    25|exact_match|↑  | 0.37|±  |0.0216|
|     |       |strict-match    |    25|exact_match|↑  | 0.37|±  |0.0216|

local-completions ({'base_url': 'http://localhost:8000/v1/completions', 'model': 'deepseek-ai/DeepSeek-V2-Lite', 'tokenized_requests': False, 'num_concurrent': 1000}), gen_kwargs: ({'temperature': 0.0}), limit: 500.0, num_fewshot: 25, batch_size: 1
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
|gsm8k|      3|flexible-extract|    25|exact_match|↑  |0.366|±  |0.0216|
|     |       |strict-match    |    25|exact_match|↑  |0.368|±  |0.0216|

local-completions ({'base_url': 'http://localhost:8000/v1/completions', 'model': 'deepseek-ai/DeepSeek-V2-Lite', 'tokenized_requests': False, 'num_concurrent': 1000}), gen_kwargs: ({'temperature': 0.0}), limit: 500.0, num_fewshot: 25, batch_size: 1
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
|gsm8k|      3|flexible-extract|    25|exact_match|↑  |0.366|±  |0.0216|
|     |       |strict-match    |    25|exact_match|↑  |0.368|±  |0.0216|
```

### DSV4-Flash



```
local-completions ({'base_url': 'http://localhost:8000/v1/completions', 'model': 'deepseek-ai/DeepSeek-V4-Flash', 'tokenized_requests': False, 'num_concurrent': 1000}), gen_kwargs: ({'temperature': 0.0}), limit: 500.0, num_fewshot: 25, batch_size: 1
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
|gsm8k|      3|flexible-extract|    25|exact_match|↑  |0.956|±  |0.0092|
|     |       |strict-match    |    25|exact_match|↑  |0.956|±  |0.0092|

local-completions ({'base_url': 'http://localhost:8000/v1/completions', 'model': 'deepseek-ai/DeepSeek-V4-Flash', 'tokenized_requests': False, 'num_concurrent': 1000}), gen_kwargs: ({'temperature': 0.0}), limit: 500.0, num_fewshot: 25, batch_size: 1
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
|gsm8k|      3|flexible-extract|    25|exact_match|↑  |0.958|±  | 0.009|
|     |       |strict-match    |    25|exact_match|↑  |0.958|±  | 0.009|

local-completions ({'base_url': 'http://localhost:8000/v1/completions', 'model': 'deepseek-ai/DeepSeek-V4-Flash', 'tokenized_requests': False, 'num_concurrent': 1000}), gen_kwargs: ({'temperature': 0.0}), limit: 500.0, num_fewshot: 25, batch_size: 1
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
|gsm8k|      3|flexible-extract|    25|exact_match|↑  |0.958|±  | 0.009|
|     |       |strict-match    |    25|exact_match|↑  |0.958|±  | 0.009|
```

### DSV3.2



```
local-completions ({'base_url': 'http://localhost:8000/v1/completions', 'model': 'deepseek-ai/DeepSeek-V3.2-Exp', 'tokenized_requests': False, 'num_concurrent': 1000}), gen_kwargs: ({'temperature': 0.0}), limit: 500.0, num_fewshot: 25, batch_size: 1
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
|gsm8k|      3|flexible-extract|    25|exact_match|↑  |0.952|±  |0.0096|
|     |       |strict-match    |    25|exact_match|↑  |0.952|±  |0.0096|

local-completions ({'base_url': 'http://localhost:8000/v1/completions', 'model': 'deepseek-ai/DeepSeek-V3.2-Exp', 'tokenized_requests': False, 'num_concurrent': 1000}), gen_kwargs: ({'temperature': 0.0}), limit: 500.0, num_fewshot: 25, batch_size: 1
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
|gsm8k|      3|flexible-extract|    25|exact_match|↑  |0.954|±  |0.0094|
|     |       |strict-match    |    25|exact_match|↑  |0.952|±  |0.0096|

local-completions ({'base_url': 'http://localhost:8000/v1/completions', 'model': 'deepseek-ai/DeepSeek-V3.2-Exp', 'tokenized_requests': False, 'num_concurrent': 1000}), gen_kwargs: ({'temperature': 0.0}), limit: 500.0, num_fewshot: 25, batch_size: 1
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
|gsm8k|      3|flexible-extract|    25|exact_match|↑  |0.946|±  |0.0101|
|     |       |strict-match    |    25|exact_match|↑  |0.946|±  |0.0101|
```

### GLM5.3



```
local-completions ({'base_url': 'http://localhost:8000/v1/completions', 'model': 'zai-org/GLM-5.3', 'tokenized_requests': False, 'num_concurrent': 1000}), gen_kwargs: ({'temperature': 0.0}), limit: 500.0, num_fewshot: 25, batch_size: 1
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
|gsm8k|      3|flexible-extract|    25|exact_match|↑  |0.922|±  | 0.012|
|     |       |strict-match    |    25|exact_match|↑  |0.922|±  | 0.012|

local-completions ({'base_url': 'http://localhost:8000/v1/completions', 'model': 'zai-org/GLM-5.3', 'tokenized_requests': False, 'num_concurrent': 1000}), gen_kwargs: ({'temperature': 0.0}), limit: 500.0, num_fewshot: 25, batch_size: 1
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
|gsm8k|      3|flexible-extract|    25|exact_match|↑  |0.946|±  |0.0101|
|     |       |strict-match    |    25|exact_match|↑  |0.946|±  |0.0101|

local-completions ({'base_url': 'http://localhost:8000/v1/completions', 'model': 'zai-org/GLM-5.3', 'tokenized_requests': False, 'num_concurrent': 1000}), gen_kwargs: ({'temperature': 0.0}), limit: 500.0, num_fewshot: 25, batch_size: 1
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
|gsm8k|      3|flexible-extract|    25|exact_match|↑  |0.928|±  |0.0116|
|     |       |strict-match    |    25|exact_match|↑  |0.928|±  |0.0116|
```

## Savings compared to main (running the lmeval workloads above)

<google-sheets-html-origin><style type="text/css"><!--td {border: 1px solid #cccccc;}br {mso-data-placement:same-cell;}--></style>

| MAX DISK SIZE = 177G                                          |                                                                           |                                                                          |
| ------------------------------------------------------------- | ------------------------------------------------------------------------- | ------------------------------------------------------------------------ |
| model                                                         | Main (FS Occupied disk size)                                              | PR (FS Occupied disk size)                                               |
| Dsv2 lite (TP=2)                                              | 70G                                                                       | 70G                                                                      |
| Dsv4-flash (TP=8)                                             | 157G                                                                      | 20G                                                                      |
| Glm5.3 (TP=8)                                                 | 177G (ENOSPC)                                                             | 103G                                                                     |
| DSV3.2 (TP=8)                                                 | 177G (ENOSPC)                                                             | 149G                                                                     |
|                                                               |                                                                           |                                                                          |
| CPU KV cache block size - Note the savings scale with TP size |                                                                           |                                                                          |
| model                                                         | Main (bytes)                                                              | PR (bytes)                                                               |
| Dsv2 lite (TP=2)                                              | 499712                                                                    | 499712                                                                   |
| Dsv4-flash (TP=8)                                             | [`4010393`](https://github.com/aarnphm/aarnphm.github.io/commit/40103936) | [`5013504`](https://github.com/aarnphm/aarnphm.github.io/commit/5013504) |
| Glm5.3 (TP=8)                                                 | [`3207168`](https://github.com/aarnphm/aarnphm.github.io/commit/32071680) | [`4009984`](https://github.com/aarnphm/aarnphm.github.io/commit/4009984) |
| DSV3.2 (TP=8)                                                 | [`2761932`](https://github.com/aarnphm/aarnphm.github.io/commit/27619328) | [`3452928`](https://github.com/aarnphm/aarnphm.github.io/commit/3452928) |

---

## comments

**varun-sundar-rabindranath** (2026-09-19T03:26:13Z):

cc @almogtavor @Etelis @bnellnm @tlrmchlsmth

**varun-sundar-rabindranath** (2026-09-19T03:30:12Z):

I have another set of evals going - will post results when done. But I think the PR is in a good state to review anyways.
Edit : Updated ✅

**varun-sundar-rabindranath** (2026-09-19T03:30:37Z):

also cc @LucasWilkinson

**mergify** (2026-09-24T07:14:30Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @varun-sundar-rabindranath.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**orozery** (2026-09-27T11:03:49Z):

/ci run

**github-actions** (2026-09-27T11:03:50Z):

✅ @varun-sundar-rabindranath, CI is now available for this PR.

- `/ci run` starts upstream CI; `/amd-ci run` starts AMD CI only.
- Your branch must contain every commit currently on its upstream target branch. Merge or rebase onto the latest target branch, then rerun the command. Append `--allow-stale` to a run command to test an outdated branch at your own risk.
- `/ci retry` retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
- `/amd-ci retry` retries failed jobs in AMD CI for the current PR head. Use `/amd-ci run` when the current head has no AMD CI build.
- `/ci cancel` cancels scheduled or running CI builds for this PR branch; `/amd-ci cancel` does the same for AMD CI only.

<!-- vllm-ci-authorized -->

**github-actions** (2026-09-27T11:04:05Z):

❌ This PR is 145 commits behind upstream `main`. Your branch must contain every commit currently on upstream `main`. No new CI build was started. Merge or rebase onto the latest `main`, then rerun `/ci run`. To test this branch at your own risk, use `/ci run --allow-stale`.

<!-- vllm-ci-command:5855267503 -->

**varun-sundar-rabindranath** (2026-09-28T13:43:27Z):

/ci run

**github-actions** (2026-09-28T13:43:48Z):

✅ Triggered [Buildkite CI #91593](https://buildkite.com/vllm/ci/builds/91593) for commit `b6f6703e5283`.

**varun-sundar-rabindranath** (2026-09-28T19:49:30Z):

> I think this might be cleaner / less brittle if add a `max_tp_shards` attribute to `AttentionSpec.max_tp_shards`; thoughts @varun-sundar-rabindranath ?
>
> like: [neuralmagic#176](https://github.com/neuralmagic/vllm/pull/176)

Thanks @LucasWilkinson . I like the idea of declaring TP sharding at the Spec level (much closer to the source and avoids checks on Spec types).
I have merged into this PR.

cc @Etelis @bnellnm Please take a look ! Thanks 🙌

**varun-sundar-rabindranath** (2026-09-28T19:49:45Z):

/ci run

**github-actions** (2026-09-28T19:50:08Z):

❌ This PR is 17 commits behind upstream `main`. Your branch must contain every commit currently on upstream `main`. No new CI build was started. Merge or rebase onto the latest `main`, then rerun `/ci run`. To test this branch at your own risk, use `/ci run --allow-stale`.

<!-- vllm-ci-command:5877297568 -->

**varun-sundar-rabindranath** (2026-09-28T19:55:11Z):

/ci run

**github-actions** (2026-09-28T19:55:33Z):

✅ Triggered [Buildkite CI #91660](https://buildkite.com/vllm/ci/builds/91660) for commit `be2521833025`.

**varun-sundar-rabindranath** (2026-09-28T22:52:50Z):

Thanks @LucasWilkinson - I ran the e2e eval test with the new changes and I can repro what is in the PR description.

**varun-sundar-rabindranath** (2026-09-29T13:26:23Z):

/ci run

**github-actions** (2026-09-29T13:26:55Z):

❌ This PR is 3 commits behind upstream `main`. Your branch must contain every commit currently on upstream `main`. No new CI build was started. Merge or rebase onto the latest `main`, then rerun `/ci run`. To test this branch at your own risk, use `/ci run --allow-stale`.

<!-- vllm-ci-command:5891243551 -->

**varun-sundar-rabindranath** (2026-09-29T17:08:15Z):

/ci run

**github-actions** (2026-09-29T17:08:38Z):

✅ Triggered [Buildkite CI #91915](https://buildkite.com/vllm/ci/builds/91915) for commit `4d9c27948f34`.

**varun-sundar-rabindranath** (2026-09-29T23:12:23Z):

Update : Having bit of a hard time trying to figure out why this test `[buildkite/ci/pr/amd-mi300-lm-eval-kv-offload](https://buildkite.com/vllm/ci/builds/91915#01a0ee85-ace4-4bfc-84a7-ad2f3cc879af)
buildkite/ci/pr/amd-mi300-lm-eval-kv-offload` <https://buildkite.com/vllm/ci/builds/91915> is failing .

The first couple of failures are real (but not sure if the test is just flaky). The tests were added yesterday with [vllm-project/vllm#50519](https://github.com/vllm-project/vllm/pull/50519) . in subsequent retries the test itself passes, but `run-amd-test.sh` reports getting a coredump with SIGABRT.

Digging into this further.

jfyi cc @AndreasKaratzas

\[Edit\]: A new retry passed the test. also, I could get the accuracy to drop on the nemotron model on `main` on H100.

**AndreasKaratzas** (2026-09-30T00:42:09Z):

@varun-sundar-rabindranath this should be fine, we are going to debug this. I see that this is a 134 exit code too.

