---
title: '58358'
pageLayout: default
slug: thoughts/inference/vllm-prs/58358
permalink: https://aarnphm.xyz/thoughts/inference/vllm-prs/58358.md
generator:
  quartz: v4.6.0
  hostedProvider: Cloudflare
  baseUrl: aarnphm.xyz
full: https://aarnphm.xyz/llms-full.txt
---
# [#58358](https://github.com/aarnphm/aarnphm.github.io/issues/58358) \[Rust Frontend\] Token-aware marker parsing for Kimi K3

[vllm-project/vllm#58358](https://github.com/vllm-project/vllm/pull/58358) · BugenZhao · MERGED · created 2026-09-23T13:06:29Z · merged 2026-10-01T04:21:52Z

## Purpose

Kimi K3 delimits channels with dedicated special tokens (`<|open|>`, `<|close|>`, `<|sep|>`, `<|end_of_msg|>`), but the model can also spell the same bytes with ordinary BPE tokens, for example while quoting an XTML snippet in its reasoning. The Rust K3 unified parser matched markers by decoded text only, so an ordinary `<|close|>think<|sep|>` inside reasoning closed the think channel early, and the rest of the message landed in the wrong channel or was dropped, with no error surfaced to the client.

This PR makes K3 channel boundaries token-aware using the per-token attribution that already reaches unified parsers through `DecodedText`: a marker’s special-token segments count as structure only when a single token with the expected ID produced them. Equal text from ordinary tokens stays content in the current channel. Text stays the parse axis, so marker words (`think`, `response`, `call`) and tag attributes are still matched by spelling.

Stacked on [#58357](https://github.com/aarnphm/aarnphm.github.io/issues/58357), which anchors a token that follows an incomplete UTF-8 byte at its own first byte. Without it, a genuine special token right after such a byte is anchored too early and would not be recognized as structural. [#58357](https://github.com/aarnphm/aarnphm.github.io/issues/58357) in turn sits on [#59048](https://github.com/aarnphm/aarnphm.github.io/issues/59048), which makes the chat roundtrip harness and the parser benchmarks feed attributed input like the serving path.

Changes:

- **Parser utils:** `Attributed<'i, M>` is a winnow stream (`Stateful<Partial<LocatingSlice<&str>>, _>`) over the buffer that carries its token anchors and the parser’s markers, so existing combinators work unchanged and grammar functions read the markers through `input.markers()` instead of an extra parameter. `Marker` is a marker spelling whose special-token segments are guarded by `SpecialToken` (spelling and ID resolved together); the same definition drives both the consuming parser and the safe-text scanner, so they cannot disagree. `MarkerStream` is implemented for `Partial<&str>` and `Attributed`, and one generic scanner serves both. Guarded markers take candidates from token anchors, so an ordinary lookalike never causes hold-back, including a partial `…<|clo` at the buffer tail. Existing text-only parsers keep passing `&[&str]` to the same helpers and have no diff; token-aware parsers call `parse_buffered_event_attributed(&DecodedText, ..)` instead of `parse_buffered_event(&str, ..)`.
- **Kimi K3:** all channel boundaries, `call` open/close, and the separator ending `call` attributes are structural. The parser requires token-attributed input, which the vLLM chat path provides; fed unattributed text, it treats every marker as content. There is no text-only fallback mode: `Marker::without_guards` keeps one cheap to add if an integrator without attribution needs it. `<|end_of_msg|>` is now required like the other structural tokens.

Known limitations, left for follow-ups:

- `argument` and `json` blocks inside a completed `call` body are still parsed from text, so a string argument that literally contains `<|close|>argument<|sep|>` is still ambiguous, as before.
- The K3 output grammar (structural tags) is still text-level, so under a constrained decode it can treat an ordinary lookalike as a boundary while the parser keeps it as content. XGrammar’s token formats can express the structural positions; that is a separate change.

Not a duplicate: open PRs touching Kimi K3 ([#57689](https://github.com/aarnphm/aarnphm.github.io/issues/57689), [#57366](https://github.com/aarnphm/aarnphm.github.io/issues/57366), [#57370](https://github.com/aarnphm/aarnphm.github.io/issues/57370), [#50291](https://github.com/aarnphm/aarnphm.github.io/issues/50291), [#53249](https://github.com/aarnphm/aarnphm.github.io/issues/53249)) address Python-side truncation, response-only streaming, and missing think-close recovery; none makes the Rust parsers token-aware.

AI assistance (Claude Code) was used for the design and implementation. I reviewed every changed line and ran the tests below.

## Test Plan

From `rust/`:



```bash
cargo test -p vllm-tokenizer -p vllm-parser -p vllm-chat --no-fail-fast
cargo clippy -p vllm-parser -p vllm-tokenizer -p vllm-chat --all-targets --features vllm-parser/test-util -- -D warnings
cargo bench -p vllm-parser --features test-util --bench kimi_k3
pre-commit run --files <changed files>
```

New coverage:

- **Marker unit tests:** accept, reject, and incomplete cases; immediate rejection of a partial lookalike; zero-width records inside a span; guardless matching; all-specials strictness; anchor versus text candidates.
- **K3 legacy suite in production mode:** every chunk is encoded with the special tokens and decoded through the real incremental decoder, and chunked cases split the attributed output at chunk boundaries.
- **K3 lookalike regressions:** an ordinary think-close or think-open inside reasoning stays reasoning; an ordinary `<|sep|>` inside `call` attributes stays attribute text; an incomplete UTF-8 byte right before a genuine close still closes; all of these hold at every char-boundary split.
- **End to end:** `roundtrip_kimi_k3` in `rust/src/chat/tests/roundtrip.rs` exercises the native renderer and parser with the real `moonshotai/Kimi-K3` tokenizer.

## Test Result

- On the stack rebased onto current `main` (with [#59048](https://github.com/aarnphm/aarnphm.github.io/issues/59048) and [#58357](https://github.com/aarnphm/aarnphm.github.io/issues/58357) below): `vllm-parser` 495, `vllm-tokenizer` 93, `vllm-chat` lib 342, `vllm-chat` integration 17, and `vllm-chat` roundtrip 20 (with `roundtrip_kimi_k3`) pass.
- Kimi K3 benchmark from [#59048](https://github.com/aarnphm/aarnphm.github.io/issues/59048), `reuse_parser`, text-only parser on [#59048](https://github.com/aarnphm/aarnphm.github.io/issues/59048) versus this PR: mixed tool calls 19.4 → 18.1 µs, long reasoning 3.22 → 3.52 ms, long string argument 401 → 315 µs. These are single runs on a shared host.
- Clippy with `-D warnings` and pre-commit pass.
- Without the [#59048](https://github.com/aarnphm/aarnphm.github.io/issues/59048) harness change, `roundtrip_kimi_k3` fails with the whole assistant message in reasoning. That is the behavior any caller feeding unattributed text would get.
- No GPU serving or accuracy eval was run. With genuine special tokens, parser output is unchanged: the legacy suite now runs in production mode and passes unchanged, and the real-tokenizer roundtrip passes. Only ordinary-token lookalikes behave differently.

---

<details>
<summary> Essential Elements of an Effective PR Description Checklist </summary>

- [x] The purpose of the PR, such as “Fix some issue (link existing issues this PR will resolve)”.
- [x] The test plan, such as providing test command.
- [x] The test results, such as pasting the results comparison before and after, or e2e results
- [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model.

</details>

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---

## comments

**BugenZhao** (2026-09-29T06:24:47Z):

/ci run

**github-actions** (2026-09-29T06:26:00Z):

✅ Triggered [Buildkite CI #91764](https://buildkite.com/vllm/ci/builds/91764) for commit `73118f21b809`.

