---
title: '57387'
pageLayout: default
slug: thoughts/inference/vllm-prs/57387
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/57387.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#57387](https://github.com/aarnphm/aarnphm.github.io/issues/57387) \[Model\] Use upstream GLM-5.3 and Qwen4-Exp configs and processor

[vllm-project/vllm#57387](https://github.com/vllm-project/vllm/pull/57387) · hmellor · MERGED · created 2026-09-17T13:52:00Z · merged 2026-10-02T02:25:19Z

## Purpose

Transformers ships `glm5_next` as of **v5.16.1** and `qwen4_exp` as of **v5.16.0**, so the vendored copies can go. Raises the `transformers` floor in `requirements/common.txt` from `>= 5.10.4` to `>= 5.16.1` — CI already pins `transformers==5.16.1` exactly in `requirements/test/{cuda,rocm,xpu}.in`, so this only makes the declared minimum match what is already tested.

Removed:

- `vllm/transformers_utils/configs/glm5_next.py` (420 lines)
- `vllm/transformers_utils/processors/glm5next.py` (953 lines)
- `vllm/transformers_utils/configs/qwen4_exp.py`
- `vllm/models/qwen4_exp/config.py`

Net **-1961 lines**.

### Not a straight deletion

The upstream classes are not drop-in replacements, so the model code moves to their spellings:

| vLLM read                  | upstream ships        |
| -------------------------- | --------------------- |
| `mla_nope`                 | `mla_use_nope`        |
| `num_experts_per_token`    | `num_experts_per_tok` |
| `moe_renormalize`          | `norm_topk_prob`      |
| `mhc_num_residual_streams` | `hc_mult`             |
| `mhc_sinkhorn_iterations`  | `hc_sinkhorn_iters`   |

`is_moe` / `is_kda_layer` / `is_linear_attn` now derive from `layer_types`, and text-level reads go through `hf_text_config` now that the top-level config no longer mirrors its `text_config` via `__getattribute__`.

I verified the rename table by loading the real `zai-org/GLM-5.3-Flash` `config.json` through upstream’s `Glm5NextConfig` and diffing the resolved attribute surface against every attribute the model tree reads.

### Behaviour deliberately preserved

Three things in the vendored code were intentional divergences, not schema, and are kept with their explanations:

1. **Vision `rms_norm_eps` stays 1e-6.** The checkpoint ships 1e-5 and upstream honours it; the vendored config force-overrode it because 1e-5 “produces repetitive/degraded image descriptions”. Now a named constant at its single call site.
2. **The frame sampler moves to `Glm5NextVideoBackend._sample_frame_indices`**, its only consumer, rather than being replaced by upstream’s `sample_frames`. See the open question below — this one is worth a second look.
3. **The `NotImplementedError` guards** for config options this implementation lacks move into `_validate_supported_config`, called from `Glm5NextModel.__init__`.

Three GLM-5.3 values have no source in either the checkpoint or the upstream config, so they become named constants rather than silently-missing attributes: `mhc_tau` (0.05), `mhc_post_mult_value` (2.0), `logit_scale` (1.0).

### Hybrid layer accounting

Both vendored configs supplied a `layers_block_type` property that mapped their sparse-attention layer type onto `"attention"`. Without it, `ModelConfig.get_num_layers_by_block_type` falls through to its `layer_types` branch, which only recognised `"full_attention"` — so both models would have counted **zero** attention layers. It now counts `deepseek_sparse_attention` and `qwen_sparse_attention` too, which is what both properties did.

`_get_full_attention_layer_indices` in the turboquant config has the same blind spot, but it already checks `layer_types` first and so is unchanged by this PR. Flagging it as a separate pre-existing issue rather than widening scope here.

### Open question: the frame sampler

This PR keeps vLLM’s own frame sampler, but I am not confident that is the right call and would like a view on it.

The sampler’s docstring justifies diverging from the reference implementation by noting that the linspace variant “cost 4 points on video grounding evals”. That comment describes only the **short-clip** branch (`total_frames < extract_t`), which requires `fps < 2`; I could not construct a case where that branch differs from upstream at all.

The branch that does differ is the common one, for an unrelated reason: vLLM walks at `1/(temporal_patch_size * target_fps)` (0.25s), collecting \~2x the candidates, then re-spreads them with `linspace`. Upstream walks at `1/target_fps` (0.5s) and keeps its picks. Over a sweep of realistic sources, **38/53 cases differ** — same frame counts, different indices:

| case           | vLLM                             | upstream                         |
| -------------- | -------------------------------- | -------------------------------- |
| 30s @30fps     | `0, 15, 30, 45, 60, 76, 91, 106` | `0, 15, 30, 45, 60, 75, 90, 105` |
| 3 frames @2fps | `0, 1, 2, 2`                     | `0, 1`                           |

vLLM’s spacing drifts off the target fps and short sources pick up a duplicated tail frame from the odd-parity pad. There is also no consistency argument for keeping our own: `create_hf_metadata` sets `do_sample_frames = len(valid_frame_indices) == source.total_frames_num`, so whenever the backend sub-samples the processor is told not to re-sample, and the backend’s choice is authoritative either way.

So switching to upstream’s `sample_frames` would delete the last vendored piece and looks more correct on the merits — but it changes frame selection for nearly every video, so it needs a video eval that I could not run. Keeping ours (what this PR does) preserves exactly what ships today at no eval cost. Happy to switch if a reviewer with a GPU wants to take the eval.

### Dropped code left over from Qwen3-Next — please sanity-check

`Qwen4ExpDecoderLayer` selected between a MoE block and a dense `Qwen3NextMLP`:



```python
mlp_only_layers = getattr(config, 'mlp_only_layers', [])
num_experts = getattr(config, 'num_experts', 0) or 0
absolute_layer_id = self.layer_idx + 1
is_moe_layer = self.layer_idx not in mlp_only_layers and (
  num_experts > 0 and absolute_layer_id % config.decoder_sparse_step == 0
)
```

This is a verbatim lift of `qwen3_next.py:479-485`, which arrived with the original support PR ([#53896](https://github.com/aarnphm/aarnphm.github.io/issues/53896)) — Qwen4Exp derives from Qwen3-Next and still subclasses `Qwen3NextAttention` and `Qwen3NextSparseMoeBlock`. `decoder_sparse_step` and `mlp_only_layers` are Qwen3-Next concepts; upstream’s `Qwen4ExpTextConfig` declares neither, and neither appears anywhere in `Qwen/Qwen3.8-Flash-Next` or `...-FP8`. They only resolved because the vendored config subclassed `Qwen3NextConfig` and inherited them, which is why nothing ever flagged the branch as dead.

Evaluating the original expression against both checkpoints with the old vendored config gives `is_moe_layer` **True for 48/48 layers**, no dense layers: with `mlp_only_layers=[]`, `num_experts=512` and `decoder_sparse_step=1` it reduces to `True` for any layer index. So this PR builds the MoE block unconditionally and drops the branch, its guards and the `Qwen3NextMLP` import (`-38/+6`).

Worth a second opinion, since it is a deletion rather than a rename: a hypothetical future checkpoint setting `mlp_only_layers` or `decoder_sparse_step` would previously have been honoured and now would not. Upstream’s `qwen4_exp` has no dense path at all (neither key appears in the model, and `validate_architecture` requires `num_experts > 0`), so such a checkpoint would not load in transformers either — which is why I think dropping it is right. Happy to restore it as `getattr(config, "decoder_sparse_step", 1)` with only the dead `else` removed if a reviewer would rather keep the guard.

## Test Plan



```bash
pytest tests/models/glm5next/test_config.py \
       tests/models/qwen4_exp/test_config.py \
       tests/transformers_utils/test_config.py \
       tests/multimodal/test_video.py \
       tests/test_config.py
pre-commit run --all-files
```

Also ran a differential check of the vendored vs upstream `smart_resize` across an 8232-point grid of sizes, frame counts and token budgets, since upstream denominates its bounds in vision tokens where the vendored one took raw pixels.

## Test Result

Every suite was run on this branch **and** on the base commit in a second worktree, because this machine has pre-existing video-codec failures.

| suite                            | base                              | branch                                |
| -------------------------------- | --------------------------------- | ------------------------------------- |
| the four files above             | 24 failed, 433 passed, 18 skipped | 24 failed, **430** passed, 18 skipped |
| `tests/multimodal/test_video.py` | 14 failed, 70 passed, 6 skipped   | 14 failed, 70 passed, 6 skipped       |

**Identical failure sets.** The −3 passed is exactly accounted for: 7 removed cases that tested the vendored config’s own behaviour, 4 added for the relocated layer-type accounting. The 24 pre-existing failures are macOS video-codec and unrelated `test_vllm_config_defaults` / `test_ir_op_priority_ctx` cases, all failing identically at the base commit.

`smart_resize` differential: **8232/8232 agree, 0 mismatches.**

`pre-commit run` passes on all changed files, including `mypy`.

Both public checkpoints were loaded through the upstream config classes to check the ported attribute surface directly, rather than inferring it:

- `Qwen/Qwen3.8-Flash-Next`: **37/39** attributes the model tree reads resolve; the two that do not are `decoder_sparse_step` and `intermediate_size`, both discussed above.
- `zai-org/GLM-5.3-Flash`: every renamed field resolves to the expected value, and the three with no source anywhere are the named constants above.

Layer accounting on the real `layer_types`, which both vendored `layers_block_type` properties used to supply:

| checkpoint         | layer_types                                           | attention | linear_attention |
| ------------------ | ----------------------------------------------------- | --------- | ---------------- |
| Qwen3.8-Flash-Next | 36 `linear_attention`, 12 `qwen_sparse_attention`     | 12        | 36               |
| GLM-5.3-Flash      | 34 `linear_attention`, 11 `deepseek_sparse_attention` | 11        | 34               |

Both sum to the full layer count. Without the `get_num_layers_by_block_type` change, both would report **0** attention layers.

### Not covered

- **No GPU here, so no model eval was run.** The change is intended to be behaviour-preserving and the value-level equivalences above were verified statically against the real checkpoint config, but a multimodal smoke test on GLM-5.3-Flash would be worth having before merge — the vision `rms_norm_eps` path in particular fails silently rather than loudly if it regresses.

## Notes

AI assistance (Claude Code) was used for this change.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---

## comments

**mergify** (2026-09-17T14:43:30Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @hmellor.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**mergify** (2026-09-18T16:08:21Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @hmellor.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**Isotr0py** (2026-09-24T16:21:46Z):

BTW, can you validate the GSM8K accuracy before merging?

**mergify** (2026-09-25T05:13:40Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @hmellor.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**hmellor** (2026-09-25T11:57:31Z):

### Model evals

GSM8K on 4xB200, this branch (`fa1dc93f27`) against its `main` parent (`09fe178dba`) in the same environment:

| model                         | eval                                                                              | base   | branch     |
| ----------------------------- | --------------------------------------------------------------------------------- | ------ | ---------- |
| `zai-org/GLM-5.3-Flash`       | `tests/evals/gsm8k/gsm8k_eval.py` (1319 questions, 5-shot, greedy)                | 0.908  | **0.911**  |
| `Qwen/Qwen3.8-Flash-Next-FP8` | `tests/evals/qwen4_exp` B200 config (EvalScope 1.10.0, thinking mode), GSM8K only | 0.9765 | **0.9788** |

Both differences are within noise (3 to 4 of 1319 questions), and Qwen clears the suite’s 0.96 floor on both. Deviations from CI: every server ran with `--gpu-memory-utilization 0.6` because the machine was shared, GLM-5.3-Flash used `--max-model-len 8192` since it has no GSM8K config of its own, and AIME25 was not run.

**hmellor** (2026-09-29T10:43:09Z):

/ci run

**github-actions** (2026-09-29T10:43:29Z):

✅ Triggered [Buildkite CI #91834](https://buildkite.com/vllm/ci/builds/91834) for commit `29b061593c72`.

**hmellor** (2026-09-29T14:14:26Z):

/ci run —allow-stale

**hmellor** (2026-09-29T15:59:48Z):

/ci run —allow-stale

**hmellor** (2026-09-29T16:22:35Z):

/ci run

**github-actions** (2026-09-29T16:23:06Z):

✅ Triggered [Buildkite CI #91902](https://buildkite.com/vllm/ci/builds/91902) for commit `2ffa283b03c9`.

**mergify** (2026-09-30T08:37:47Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @hmellor.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**hmellor** (2026-09-30T08:41:11Z):

/ci run

**github-actions** (2026-09-30T08:42:00Z):

✅ Triggered [Buildkite CI #92092](https://buildkite.com/vllm/ci/builds/92092) for commit `39cead38c541`.

**hmellor** (2026-09-30T09:33:07Z):

/ci retry

**github-actions** (2026-09-30T09:33:28Z):

✅ Queued 2 failed job(s) for retry in [Buildkite CI #92092](https://buildkite.com/vllm/ci/builds/92092).

**mergify** (2026-09-30T11:23:06Z):

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @hmellor.

<https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork>

**hmellor** (2026-09-30T11:38:52Z):

/ci run

**github-actions** (2026-09-30T11:39:14Z):

✅ Triggered [Buildkite CI #92124](https://buildkite.com/vllm/ci/builds/92124) for commit `f26e1b8a4c6a`.

**hmellor** (2026-10-01T15:11:26Z):

/ci run

**github-actions** (2026-10-01T15:11:47Z):

❌ This PR is 1 commit behind upstream `main`. Your branch must contain every commit currently on upstream `main`. No new CI build was started. Merge or rebase onto the latest `main`, then rerun `/ci run`. To test this branch at your own risk, use `/ci run --allow-stale`.

<!-- vllm-ci-command:5934315094 -->

**hmellor** (2026-10-01T15:17:00Z):

/ci run

**github-actions** (2026-10-01T15:17:24Z):

✅ Triggered [Buildkite CI #92427](https://buildkite.com/vllm/ci/builds/92427) for commit `e133c9c05a5d`.

**hmellor** (2026-10-01T18:34:34Z):

/ci retry

**github-actions** (2026-10-01T18:34:55Z):

✅ Queued 70 failed job(s) for retry in [Buildkite CI #92427](https://buildkite.com/vllm/ci/builds/92427).

**hmellor** (2026-10-01T19:07:13Z):

/ci retry

**github-actions** (2026-10-01T19:07:35Z):

✅ Queued 2 failed job(s) for retry in [Buildkite CI #92427](https://buildkite.com/vllm/ci/builds/92427).

**hmellor** (2026-10-01T21:52:14Z):

/ci run

**github-actions** (2026-10-01T21:52:34Z):

✅ Triggered [Buildkite CI #92498](https://buildkite.com/vllm/ci/builds/92498) for commit `7c83f2caed45`.

