[WIP] Add GLM-5.2-NVFP4 GB300 SGLang AgentX recipes - #2608
Conversation
GB300 Dynamo-SGLang agentic-coding sweep for GLM-5.2-NVFP4, 7 points across two config keys: - glm5.2-fp4-gb300-dynamo-sglang-agentic-agg: aggregated single-worker TP8 (conc 2/4) - glm5.2-fp4-gb300-dynamo-sglang-agentic-disagg: low-latency 1P2D/1P4D/1P6D TP4-prefill/TP4-decode (conc 48/48/45) and high-throughput 2P1D DEP8-prefill/DEP16-decode (conc 128/192)
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| gpus_per_node: 4 | ||
| frontend: |
There was a problem hiding this comment.
🔴 glm5.2-agentx.yaml sets nginx_session_affinity_header: X-Correlation-ID (line 13) while also setting AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: "true" (line 164), which per benchmark_lib.sh makes aiperf skip conv-aware routing and instead send an X-Dynamo-Session-ID header — so nginx keys affinity on a header aiperf never sends in this mode. With enable_multiple_frontends spreading requests across 5 dynamo frontends, this silently breaks multi-turn session pinning and skews the throughput/latency numbers. Fix: change the header to X-Dynamo-Session-ID, matching all 13 other recipes (deepseek-v4 and qwen3.5 gb300-fp4 agentic) that pair this env var with nginx affinity.
Extended reasoning...
The bug: glm5.2-agentx.yaml sets both frontend.nginx_session_affinity_header: X-Correlation-ID (line 13) and benchmark.env.AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: "true" (line 164). These two settings are mutually incompatible: the env var changes which header aiperf actually sends for session routing, and nginx ends up configured to key affinity on the wrong one.
The mechanism, traced through benchmarks/benchmark_lib.sh:2043-2055: the block that appends --use-dynamo-conv-aware-routing to the aiperf replay command (the mode that relies on X-Correlation-ID via nvext.session_control) is gated on AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID != "true". Since this recipe sets that var to "true", the flag is never added. The comment at line 2050-2051 spells out the alternative explicitly: this env var makes aiperf 'route by X-Dynamo-Session-ID header,' and X-Correlation-ID is 'useful tracing metadata but does not establish that binding by itself.' So in this recipe's actual runtime mode, aiperf sends X-Dynamo-Session-ID for routing — not X-Correlation-ID.
Why nothing catches this: nginx session affinity is a pure routing config — there's no validation tying the configured header to what the benchmark client actually emits. The mismatch produces no error; nginx just hashes on a header value that's either absent or unrelated to the conversation, and the request routes inconsistently across frontends.
Why existing recipes don't have this problem: every other recipe that pairs AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: "true" with nginx_session_affinity sets the header to X-Dynamo-Session-ID — confirmed via grep across all 13 deepseek-v4 and qwen3.5 gb300-fp4 agentic recipes in srt-slurm-recipes/. This new file and one pre-existing file (sglang/glm5.2/agentic/disagg-h200-2p2d-pcp8-tp8-dp8-mtp.yaml, out of scope for this PR) are the only two with X-Correlation-ID, and this PR repeats the same mistake in a brand-new file rather than following the validated pattern.
Impact: this recipe uses zip_override_mtp_agentx_hightpt/zip_override_mtp_agentx_lowlat, both of which set enable_multiple_frontends: true with num_additional_frontends: 4, so nginx load-balances across 5 dynamo frontends. Session affinity exists specifically to pin each multi-turn agentic conversation to one frontend across turns so its KV-prefix cache stays warm. With affinity keyed on a header aiperf doesn't send for routing in this mode, conversation turns scatter across frontends, degrading KV-prefix-cache hit rate and silently skewing the throughput/latency numbers this entire 7-point sweep exists to measure — with no error raised anywhere in the pipeline.
Step-by-step proof:
- Recipe sets
AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: "true"inbenchmark.env(line 164). agentic_srt.shsourcesbenchmark_lib.sh, which builds the aiperf replay command.- At
benchmark_lib.sh:2052-2054, the guard... && "${AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID:-false}" != "true"evaluates false, so--use-dynamo-conv-aware-routing(theX-Correlation-ID-based path) is never added toREPLAY_CMD. - Per the code comment at line 2050-2051, aiperf instead routes using an
X-Dynamo-Session-IDheader derived from the correlation ID. - Meanwhile nginx (configured at line 13) hashes/pins on
X-Correlation-ID, a header that plays no role in aiperf's actual routing/session-binding behavior in this mode. - Result: nginx's sticky routing is decoupled from aiperf's real session identity, and with 5 frontends behind it, conversation turns are not reliably pinned to the same frontend.
The fix is a one-line change: nginx_session_affinity_header: X-Dynamo-Session-ID, aligning with every other recipe using this env var.
| # Aggregated: one worker serves both prefill and decode, so GPU accounting | ||
| # is the single-worker form num_gpus = TP (not the disagg prefill+decode | ||
| # sum). Force the single-node post-processing path in | ||
| # process_agentic_result.py; TP must match the aggregated tensor-parallel-size. | ||
| IS_MULTINODE: "false" | ||
| TP: "8" | ||
| AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: "true" |
There was a problem hiding this comment.
🟡 glm5.2-agentx-agg.yaml sets IS_MULTINODE: "false" (lines 100-106) even though the recipe spans 2 physical GB300 nodes (agg_nodes: 2, TP8). This makes process_agentic_result.py._gpu_shape() write the single-node result schema (is_multinode=false, no prefill/decode worker-breakdown fields) for a physically multi-node run, diverging from the existing precedent for this exact topology (vllm deepseek-v4's agg-gb300-tp8-mtp-agentic.yaml, which sets IS_MULTINODE: "true" for the identical agg_nodes:2/TP8 shape). The GPU count/throughput numbers themselves are unaffected (both schema paths compute num_gpus=8, tp=8), so this is a metadata/schema-labeling inconsistency rather than a numeric error — worth aligning to IS_MULTINODE: "true" to match the established convention, but not blocking.
Extended reasoning...
What the bug is: glm5.2-agentx-agg.yaml declares resources.agg_nodes: 2 with gpus_per_agg: 8 (TP8) — i.e. a single aggregated SGLang worker whose 8 GPUs are spread across 2 physical GB300 nodes (gpus_per_node: 4). Despite this, benchmark.env sets IS_MULTINODE: "false" (line 104), with a comment claiming this is needed to 'force the single-node post-processing path' so that num_gpus = TP rather than the disagg prefill+decode sum.
Where it manifests: process_agentic_result.py._gpu_shape() branches on IS_MULTINODE when building the ingested result record. The false path emits only {pp, dcp_size, pcp_size} and derives num_gpus = tp * pp * pcp_size; the true path emits the full prefill/decode worker-breakdown (prefill_num_workers, decode_num_workers, num_prefill_gpu, num_decode_gpu). The chosen false value causes the ingested record to be tagged is_multinode=false and to omit the worker-breakdown fields, even though the run is physically 2-node.
Why existing conventions don't catch this: every other sglang recipe using IS_MULTINODE: "false" (deepseek-v4 agg-gb300-tp4-mtp-kvoffload.yaml, the qwen3.5 agg-gb300-tp2 recipes) is genuinely single-node (agg_nodes: 1), so nothing previously exercised this branch for a multi-node aggregated worker. The one existing precedent for this exact topology — an aggregated TP8 worker spanning 2 nodes — is vllm deepseek-v4's agg-gb300-tp8-mtp-agentic.yaml, which deliberately sets IS_MULTINODE: "true" with an explicit comment: 'Keep aggregate workers in the multinode result schema so ingestion uses the zero decode-worker count instead of duplicating TP into P and D.' GLM-5.2's agg recipe copies the single-node pattern instead, making it the first sglang aggregated recipe to actually span >1 node while using the single-node flag.
Step-by-step proof of the divergence (not a numeric bug, but a schema/labeling one): For this recipe, agg_workers=1, decode num-worker=0. Under IS_MULTINODE=false (as written): num_gpus = tp*pp*pcp_size = 8*1*1 = 8, tp=8, no worker-breakdown fields recorded. Under IS_MULTINODE=true (the vllm-precedent path): since num_decode_gpu <= 0, the decode side is zeroed (process_agentic_result.py lines ~155-160), giving num_prefill_gpu=1*8=8, num_decode_gpu=0, num_gpus=8, tp=8 — identical GPU accounting — but additionally recording prefill_num_workers/decode_num_workers and matching the schema used by every other multi-node aggregated recipe. So the recipe author's stated rationale (avoid duplicating TP into P+D) is equally satisfied by the multinode path; IS_MULTINODE: "false" isn't required to get correct GPU counts.
Impact: No crash and no incorrect throughput/GPU-count numbers — both code paths agree on num_gpus=8, tp=8 for this specific config. The impact is that the ingested JSON for every agg data point in this sweep records is_multinode=false and a different/incomplete field set than the established convention for a 2-node aggregated worker (and than the master config's own multinode: true declaration for this config key, which drives sweep/job allocation separately from result-schema selection). Downstream consumers that key off is_multinode (e.g. reuse-validation logic keying 'single' vs 'multi') will place this run in the single-node namespace despite it being multi-node.
Suggested fix: Set IS_MULTINODE: "true" in the agg recipe (matching the vllm agg-gb300-tp8-mtp-agentic.yaml precedent for the identical topology) and drop/update the comment claiming the single-node path is required, since the multinode path produces the same GPU accounting while recording accurate topology metadata.
On the refutation: One verifier argued this is intentional and that all sglang agg recipes use IS_MULTINODE=false by convention. That's true only for genuinely single-node sglang agg recipes — the sole existing example of a multi-node aggregated worker (the vllm sibling with the identical agg_nodes:2/TP8 shape) explicitly uses true, precisely because it produces the same numbers with better metadata. So this recipe is inconsistent with the closest applicable precedent, not consistent with a broader rule.
| glm5.2-fp4-gb300-dynamo-sglang-agentic-agg: | ||
| image: lmsysorg/sglang:nightly-dev-cu13-20260805-211ee642 | ||
| model: nvidia/GLM-5.2-NVFP4 | ||
| model-prefix: glm5.2 | ||
| runner: cluster:gb300-nv | ||
| precision: fp4 | ||
| framework: dynamo-sglang | ||
| router: { name: dynamo-router, version: "71eb001e17fa73c742f0afe1a6ed96836cb135fd" } | ||
| multinode: true | ||
| disagg: false | ||
| scenarios: | ||
| agentic-coding: | ||
| - dram-utilization: 0.80 | ||
| search-space: | ||
| # aggregated 1-worker TP8 (prefill+decode on the same GPUs) | ||
| - spec-decoding: mtp | ||
| conc-list: [2, 4] | ||
| kv-offloading: none | ||
| prefill: | ||
| num-worker: 1 | ||
| tp: 8 | ||
| ep: 1 | ||
| dp-attn: false | ||
| additional-settings: |
There was a problem hiding this comment.
🟡 Both new GB300 recipes enable SGLang's hierarchical/DRAM KV cache (enable-hierarchical-cache: true, hicache-size: 135, etc. in glm5.2-agentx-agg.yaml and the prefill config of glm5.2-agentx.yaml), but every search-space point for glm5.2-fp4-gb300-dynamo-sglang-agentic-agg/-disagg in configs/nvidia-master.yaml (lines 8087-8110) sets kv-offloading: none with no backend. This should be kv-offloading: dram, kv-offload-backend: {name: hicache} to match the repo-wide convention (e.g. the same-model glm5.2-fp4-b200-sglang-agentic-mtp entry just above it), otherwise the published benchmark metadata will misrepresent these 7 runs as not using KV offloading.
Extended reasoning...
The bug: Both new recipe files enable SGLang's hierarchical KV cache (HiCache) — a host-DRAM offload feature for the KV cache pool:
glm5.2-agentx-agg.yaml(aggregatedsglang_config):enable-hierarchical-cache: true,hicache-write-policy: write_back,hicache-size: 135,hicache-io-backend: directglm5.2-agentx.yamlprefillsglang_config: identical settings
Yet in configs/nvidia-master.yaml:8087-8110, every one of the 7 search-space points across both new config keys (glm5.2-fp4-gb300-dynamo-sglang-agentic-agg and -disagg) sets kv-offloading: none with no kv-offload-backend specified.
Why this is wrong: The repository has an established, consistently-followed convention: any recipe pairing that sets enable-hierarchical-cache: true labels the sweep entry kv-offloading: dram, kv-offload-backend: {name: hicache}. This is true across the board — the deepseek-v4 *-kvoffload.yaml recipes, the qwen3.5 GB300 agentic hicache recipes, and most tellingly, the same-model sibling entry directly above this one in the same file: glm5.2-fp4-b200-sglang-agentic-mtp at line ~8085 sets kv-offloading: dram, kv-offload-backend: { name: hicache } for a hicache config, with a comment explicitly describing it as "HiCache host-DRAM offload." The new GB300 entries enable the identical underlying feature but deviate from that labeling.
Where this matters downstream: KV_OFFLOADING is a required environment variable consumed by utils/agentic/aggregation/process_agentic_result.py (_validate_kv_offload_env), which validates it against KV_OFFLOAD_BACKEND and writes it verbatim into the aggregated result row's kv_offloading field. Concretely: for each of these 7 sweep points, the pipeline will read kv-offloading: none from the config, pass KV_OFFLOADING=none with no KV_OFFLOAD_BACKEND (which passes validation since none + empty backend is a valid combination), and stamp the published benchmark row with kv_offloading: none — even though the run is actually using a 135GB host-DRAM hierarchical KV cache with write_back policy.
Impact: This doesn't cause a crash or fail the sweep — none + empty backend is a self-consistent (if incorrect) combination that passes _validate_kv_offload_env. The impact is purely in the published benchmark metadata: these 7 GLM-5.2 GB300 AgentX results will be mislabeled as not using KV offloading, which breaks apples-to-apples comparison against the sibling glm5.2-fp4-b200-sglang-agentic-mtp entry and other hicache-enabled recipes in the aggregated dataset, and misrepresents the actual serving configuration to anyone consuming the results.
Fix: In configs/nvidia-master.yaml, change all kv-offloading: none entries under both glm5.2-fp4-gb300-dynamo-sglang-agentic-agg and glm5.2-fp4-gb300-dynamo-sglang-agentic-disagg to kv-offloading: dram, kv-offload-backend: { name: hicache }, matching the sibling B200 entry's convention.
| glm5.2-fp4-gb300-dynamo-sglang-agentic-agg: | ||
| image: lmsysorg/sglang:nightly-dev-cu13-20260805-211ee642 | ||
| model: nvidia/GLM-5.2-NVFP4 |
There was a problem hiding this comment.
🟡 Two pre-existing comments (nvidia-master.yaml:8031-8033, :8064-8066) claim the v0.5.16-cu130 image bump 'is the tag the GLM-5.2 GB300 dynamo-sglang agentic recipes already run.' This PR adds exactly those recipes (glm5.2-fp4-gb300-dynamo-sglang-agentic-agg/-disagg) with image lmsysorg/sglang:nightly-dev-cu13-20260805-211ee642 instead, so both comments are now factually wrong. Worth a quick follow-up fix to either comment (mention the nightly split) so future readers don't assume image parity across the three GLM-5.2 AgentX SKUs.
Extended reasoning...
What the bug is. Two adjacent, pre-existing comments in configs/nvidia-master.yaml make a cross-referencing claim about image provenance that this PR breaks:
- Lines 8025-8033 (comment above
glm5.2-fp4-b300-sglang-agentic-mtp): 'Image is bumped to v0.5.16-cu130: it is the first release that reads SGLANG_SIMULATE_ACC_TOKEN_MODE, and it is the tag the GLM-5.2 GB300 dynamo-sglang agentic recipes already run.' - Lines 8059-8066 (comment above
glm5.2-fp4-b200-sglang-agentic-mtp): 'Image lmsysorg/sglang:v0.5.16-cu130 is the first release that reads SGLANG_SIMULATE_ACC_TOKEN_MODE and is what the B300 sibling and the GLM-5.2 GB300 dynamo-sglang agentic recipes run.'
Both comments explicitly distinguish the 'B300 sibling' (glm5.2-fp4-b300-sglang-agentic-mtp, a single-node sglang framework recipe) from 'the GLM-5.2 GB300 dynamo-sglang agentic recipes' — a distinct, separately-named category. Before this PR, no such dynamo-sglang GB300 GLM-5.2 recipe existed in the config, so the claim was forward-looking/unverifiable. This PR adds exactly that category: glm5.2-fp4-gb300-dynamo-sglang-agentic-agg and glm5.2-fp4-gb300-dynamo-sglang-agentic-disagg (lines 8087+), both with framework: dynamo-sglang, runner: cluster:gb300-nv.
The contradiction. Both new recipes pin image: lmsysorg/sglang:nightly-dev-cu13-20260805-211ee642 (line 8088 and its -disagg sibling), not v0.5.16-cu130. So the two pre-existing comments' claim is now directly falsified by the very recipes they name — there is no other GLM-5.2 GB300 dynamo-sglang agentic recipe in the file that could make the comments true.
Step-by-step proof:
- Read comment at line 8031-8033: claims v0.5.16-cu130 'is the tag the GLM-5.2 GB300 dynamo-sglang agentic recipes already run.'
- Search the file for recipes matching 'GB300' + 'dynamo-sglang' + 'agentic' + 'GLM-5.2' — the only matches are
glm5.2-fp4-gb300-dynamo-sglang-agentic-aggand-disagg, both added by this PR. - Check their
image:field:lmsysorg/sglang:nightly-dev-cu13-20260805-211ee642. - Compare to the claimed tag:
v0.5.16-cu130. These are different, unrelated tags (nightly dev snapshot vs. a numbered release) — not equivalent. - Conclusion: the comment's factual claim is false as of this PR.
Why nothing catches this. These are free-text comments, not data — there's no schema or CI check that cross-validates a comment's image-tag claim against sibling recipes' actual image: fields. Nothing prevented the PR from merging with this now-stale text.
Impact. Purely a documentation/comment accuracy issue — no runtime effect, since the comments only annotate the unrelated b300-sglang-agentic-mtp/b200-sglang-agentic-mtp recipes, whose own image: fields are unaffected. The risk is confusing a future reader who cross-references SGLANG_SIMULATE_ACC_TOKEN_MODE support across the three GLM-5.2 AgentX SKUs and wrongly assumes all three share an image tag.
Suggested fix. Update the two comments to note the GB300 dynamo-sglang agentic recipes run a nightly image (nightly-dev-cu13-20260805-211ee642) rather than v0.5.16-cu130, or drop the cross-reference to those recipes entirely if it's no longer load-bearing for explaining the b300/b200 image choice.
The agentic CI applies the agg config-key with CONFIG_FILE=<recipe> and no zip_override selector, so srtctl apply expects the flat single-variant recipe schema (top-level name/model/backend), not a base:-wrapped base+override recipe. The disagg recipe is applied with a :zip_override selector and is unaffected.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31816819527 |
- disagg frontend: nginx session-affinity header X-Correlation-ID -> X-Dynamo-Session-ID (aiperf sends X-Dynamo-Session-ID for routing when AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID=true, so nginx affinity must key on it), session-affinity ttl 1200 -> 3600, drop DYN_ROUTER_TEMPERATURE so router-mode kv stays KV-aware, and move enable_multiple_frontends to base (default node-capped count) instead of a fixed 4 per override - disagg hightpt: drop the enable-flashinfer-allreduce-fusion:false override so it inherits the base decode's true - agg recipe: IS_MULTINODE true for the 2-node aggregated worker (decode num-worker 0 keeps num_gpus=8 via the multinode result schema); drop unused TP - config: kv-offloading none -> dram + kv-offload-backend hicache for all 7 GLM-5.2 GB300 points (recipes run HiCache), matching the B200 sibling - config: correct two stale comments that named the GB300 dynamo-sglang agentic recipes as running v0.5.16-cu130 (they run a nightly image)
…gentx # Conflicts: # perf-changelog.yaml
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31820906328 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31825558603 |
|
/reuse-sweep-run |
2 participants
GB300 Dynamo-SGLang agentic-coding sweep for GLM-5.2-NVFP4, 7 points across two config keys: