Skip to content

Add GLM-5.2-NVFP4 GB300 SGLang AgentX recipes / 新增 GLM-5.2-NVFP4 GB300 SGLang AgentX 配方 #2608

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
csahithi wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Add GLM-5.2-NVFP4 GB300 SGLang AgentX recipes / 新增 GLM-5.2-NVFP4 GB300 SGLang AgentX 配方 #2608

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
c75353f
Add GLM-5.2-NVFP4 GB300 AgentX agentic benchmarks (agg + disagg)
csahithi Aug 14, 2026
819995f
Merge remote-tracking branch 'origin/main' into nv-glm5.2-fp4-gb300-a…
csahithi Aug 14, 2026
b461fd9
changelog: point GLM-5.2 GB300 AgentX entry at PR #2608
csahithi Aug 14, 2026
8c414bb
Flatten GLM-5.2 GB300 agg recipe to the selector-free agentic schema
csahithi Aug 14, 2026
cc1e0d2
Address PR review feedback on GLM-5.2 GB300 AgentX
csahithi Aug 14, 2026
b384c40
Merge remote-tracking branch 'origin/main' into nv-glm5.2-fp4-gb300-a…
csahithi Aug 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
118 changes: 118 additions & 0 deletions ...marks/multi_node/srt-slurm-recipes/sglang/glm5.2/gb300-fp4/agentic/glm5.2-agentx-agg.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
# Agentic-coding SGLang aggregated recipe for GLM-5.2-NVFP4 on GB300
# (single aggregated worker, TP8, EAGLE MTP + hierarchical-cache KV).
#
# Flat single-variant schema the agentic CI flow expects: applied via
# CONFIG_FILE=<this file> with no zip_override selector, so it must NOT be
# base:-wrapped. Concurrency is not a recipe field; the GHA matrix fans out
# one job per concurrency from the master-config conc-list into agentic_srt.sh.
name: gb300-fp4-glm5.2-agentx-agg
model:
path: glm-5.2-fp4
container: dynamo-sglang
precision: fp4
resources:
gpu_type: gb300
gpus_per_node: 4
agg_nodes: 2
agg_workers: 1
gpus_per_agg: 8
frontend:
type: dynamo
env:
PIP_BREAK_SYSTEM_PACKAGES: '1'
args:
router-mode: kv
active-decode-blocks-threshold: None
active-prefill-tokens-threshold: None
active-prefill-tokens-threshold-frac: None
router-session-affinity-ttl-secs: 3600
dynamo:
hash: 71eb001e17fa73c742f0afe1a6ed96836cb135fd
install: true
backend:
type: sglang
sglang_config:
aggregated:
served-model-name: nvidia/GLM-5.2-NVFP4
trust-remote-code: true
quantization: modelopt_fp4
kv-cache-dtype: fp8_e4m3
tensor-parallel-size: 8
data-parallel-size: 1
expert-parallel-size: 1
enable-dp-attention: false
enable-dp-lm-head: false
max-running-requests: 10
cuda-graph-max-bs: 10
chunked-prefill-size: 8192
max-prefill-tokens: 8192
context-length: 1048576
speculative-algorithm: EAGLE
speculative-num-steps: 4
speculative-eagle-topk: 1
speculative-num-draft-tokens: 5
nsa-prefill-backend: trtllm
nsa-decode-backend: trtllm
moe-runner-backend: flashinfer_trtllm
fp4-gemm-backend: flashinfer_trtllm
disable-shared-experts-fusion: true
enable-flashinfer-allreduce-fusion: true
weight-loader-prefetch-checkpoints: true
model-loader-extra-config: '{"enable_multithread_load": true}'
mem-fraction-static: 0.8
enable-hierarchical-cache: true
hicache-write-policy: write_back
hicache-size: 135
hicache-io-backend: direct
enable-metrics: true
enable-cache-report: true
aggregated_environment:
TORCH_DISTRIBUTED_DEFAULT_TIMEOUT: '1800'
PYTHONUNBUFFERED: '1'
DYN_SKIP_SGLANG_LOG_FORMATTING: '1'
MC_TE_METRIC: 'true'
NCCL_CUMEM_ENABLE: '1'
SGLANG_USE_MESSAGE_QUEUE_BROADCASTER: '0'
SGLANG_DISABLE_TP_MEMORY_INBALANCE_CHECK: '1'
SGLANG_ENABLE_THINKING: '1'
SGLANG_REASONING_EFFORT: max
SGLANG_ENABLE_UNIFIED_RADIX_TREE: '1'
SGLANG_HICACHE_DEBUG_LOG: '1'
SGLANG_HICACHE_DEBUG_SAMPLE_RATE: '16384'
SGLANG_MOE_NVFP4_DISPATCH: '1'
SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE: '1'
SGLANG_CLIP_MAX_NEW_TOKENS_ESTIMATION: '8'
SGLANG_SIMULATE_ACC_LEN: '3.33'
SGLANG_SIMULATE_ACC_METHOD: match-expected
SGLANG_SIMULATE_ACC_TOKEN_MODE: real-draft-token
PIP_BREAK_SYSTEM_PACKAGES: '1'
SGLANG_DG_CACHE_DIR: /deepgemm_cache
FLASHINFER_WORKSPACE_BASE: /flashinfer_cache
MC_FORCE_MNNVL: '1'
NCCL_MNNVL_ENABLE: '1'
SGLANG_MOONCAKE_CUSTOM_MEM_POOL: 'True'
UCX_TLS: ib,cuda_copy,cuda_ipc,sm,self
NVSHMEM_REMOTE_TRANSPORT: none
health_check:
max_attempts: 1440
interval_seconds: 10
benchmark:
type: custom
command: bash /infmax-workspace/benchmarks/multi_node/agentic_srt.sh
env:
INFMAX_CONTAINER_WORKSPACE: /infmax-workspace
RESULT_DIR: /logs/agentic
PORT: "8000"
IS_MULTINODE: "true"
AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: "true"
AIPERF_DATASET_MMAP_CACHE_DIR: /aiperf_mmap_cache
HF_HUB_CACHE: /hf_hub_cache
infra:
etcd_nats_dedicated_node: false
nats_max_payload_mb: 64
sbatch_directives:
mem: '0'
cpus-per-task: '144'
srun_options:
mem: '0'
container-remap-root: ''
219 changes: 219 additions & 0 deletions benchmarks/multi_node/srt-slurm-recipes/sglang/glm5.2/gb300-fp4/agentic/glm5.2-agentx.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,219 @@
base:
name: gb300-fp4-glm5.2-agentx
model:
path: glm-5.2-fp4
container: dynamo-sglang
precision: fp4
resources:
gpu_type: gb300
gpus_per_node: 4
frontend:
Comment on lines +9 to +10

@claude claude Bot Aug 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 glm5.2-agentx.yaml sets nginx_session_affinity_header: X-Correlation-ID (line 13) while also setting AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: "true" (line 164), which per benchmark_lib.sh makes aiperf skip conv-aware routing and instead send an X-Dynamo-Session-ID header — so nginx keys affinity on a header aiperf never sends in this mode. With enable_multiple_frontends spreading requests across 5 dynamo frontends, this silently breaks multi-turn session pinning and skews the throughput/latency numbers. Fix: change the header to X-Dynamo-Session-ID, matching all 13 other recipes (deepseek-v4 and qwen3.5 gb300-fp4 agentic) that pair this env var with nginx affinity.

Extended reasoning...

The bug: glm5.2-agentx.yaml sets both frontend.nginx_session_affinity_header: X-Correlation-ID (line 13) and benchmark.env.AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: "true" (line 164). These two settings are mutually incompatible: the env var changes which header aiperf actually sends for session routing, and nginx ends up configured to key affinity on the wrong one.

The mechanism, traced through benchmarks/benchmark_lib.sh:2043-2055: the block that appends --use-dynamo-conv-aware-routing to the aiperf replay command (the mode that relies on X-Correlation-ID via nvext.session_control) is gated on AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID != "true". Since this recipe sets that var to "true", the flag is never added. The comment at line 2050-2051 spells out the alternative explicitly: this env var makes aiperf 'route by X-Dynamo-Session-ID header,' and X-Correlation-ID is 'useful tracing metadata but does not establish that binding by itself.' So in this recipe's actual runtime mode, aiperf sends X-Dynamo-Session-ID for routing — not X-Correlation-ID.

Why nothing catches this: nginx session affinity is a pure routing config — there's no validation tying the configured header to what the benchmark client actually emits. The mismatch produces no error; nginx just hashes on a header value that's either absent or unrelated to the conversation, and the request routes inconsistently across frontends.

Why existing recipes don't have this problem: every other recipe that pairs AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: "true" with nginx_session_affinity sets the header to X-Dynamo-Session-ID — confirmed via grep across all 13 deepseek-v4 and qwen3.5 gb300-fp4 agentic recipes in srt-slurm-recipes/. This new file and one pre-existing file (sglang/glm5.2/agentic/disagg-h200-2p2d-pcp8-tp8-dp8-mtp.yaml, out of scope for this PR) are the only two with X-Correlation-ID, and this PR repeats the same mistake in a brand-new file rather than following the validated pattern.

Impact: this recipe uses zip_override_mtp_agentx_hightpt/zip_override_mtp_agentx_lowlat, both of which set enable_multiple_frontends: true with num_additional_frontends: 4, so nginx load-balances across 5 dynamo frontends. Session affinity exists specifically to pin each multi-turn agentic conversation to one frontend across turns so its KV-prefix cache stays warm. With affinity keyed on a header aiperf doesn't send for routing in this mode, conversation turns scatter across frontends, degrading KV-prefix-cache hit rate and silently skewing the throughput/latency numbers this entire 7-point sweep exists to measure — with no error raised anywhere in the pipeline.

Step-by-step proof:

  1. Recipe sets AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: "true" in benchmark.env (line 164).
  2. agentic_srt.sh sources benchmark_lib.sh, which builds the aiperf replay command.
  3. At benchmark_lib.sh:2052-2054, the guard ... && "${AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID:-false}" != "true" evaluates false, so --use-dynamo-conv-aware-routing (the X-Correlation-ID-based path) is never added to REPLAY_CMD.
  4. Per the code comment at line 2050-2051, aiperf instead routes using an X-Dynamo-Session-ID header derived from the correlation ID.
  5. Meanwhile nginx (configured at line 13) hashes/pins on X-Correlation-ID, a header that plays no role in aiperf's actual routing/session-binding behavior in this mode.
  6. Result: nginx's sticky routing is decoupled from aiperf's real session identity, and with 5 frontends behind it, conversation turns are not reliably pinned to the same frontend.

The fix is a one-line change: nginx_session_affinity_header: X-Dynamo-Session-ID, aligning with every other recipe using this env var.

type: dynamo
nginx_session_affinity: true
nginx_session_affinity_header: X-Dynamo-Session-ID
enable_multiple_frontends: true
env:
PIP_BREAK_SYSTEM_PACKAGES: '1'
args:
router-mode: kv
active-decode-blocks-threshold: None
active-prefill-tokens-threshold: None
active-prefill-tokens-threshold-frac: None
router-session-affinity-ttl-secs: 3600
dynamo:
hash: 71eb001e17fa73c742f0afe1a6ed96836cb135fd
install: true
backend:
type: sglang
prefill_environment:
TORCH_DISTRIBUTED_DEFAULT_TIMEOUT: '1800'
PYTHONUNBUFFERED: '1'
DYN_SKIP_SGLANG_LOG_FORMATTING: '1'
SGLANG_DISAGGREGATION_HEARTBEAT_MAX_FAILURE: '100000'
SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT: '100000'
SGLANG_DISAGGREGATION_WAITING_TIMEOUT: '100000'
MC_TE_METRIC: 'true'
MC_FORCE_MNNVL: '1'
NCCL_MNNVL_ENABLE: '1'
NCCL_CUMEM_ENABLE: '1'
SGLANG_MOONCAKE_CUSTOM_MEM_POOL: 'True'
SGLANG_USE_MESSAGE_QUEUE_BROADCASTER: '0'
SGLANG_DISABLE_TP_MEMORY_INBALANCE_CHECK: '1'
SGLANG_ENABLE_THINKING: '1'
SGLANG_ENABLE_UNIFIED_RADIX_TREE: '1'
SGLANG_HICACHE_DEBUG_LOG: '1'
SGLANG_HICACHE_DEBUG_SAMPLE_RATE: '16384'
SGLANG_REASONING_EFFORT: max
PIP_BREAK_SYSTEM_PACKAGES: '1'
UCX_TLS: ib,cuda_copy,cuda_ipc,sm,self
NVSHMEM_REMOTE_TRANSPORT: none
SGLANG_DG_CACHE_DIR: /deepgemm_cache
FLASHINFER_WORKSPACE_BASE: /flashinfer_cache
SGLANG_SIMULATE_ACC_LEN: '2.5'
decode_environment:
TORCH_DISTRIBUTED_DEFAULT_TIMEOUT: '1800'
PYTHONUNBUFFERED: '1'
DYN_SKIP_SGLANG_LOG_FORMATTING: '1'
SGLANG_DISAGGREGATION_HEARTBEAT_MAX_FAILURE: '100000'
SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT: '100000'
SGLANG_DISAGGREGATION_WAITING_TIMEOUT: '100000'
MC_TE_METRIC: 'true'
MC_FORCE_MNNVL: '1'
NCCL_MNNVL_ENABLE: '1'
NCCL_CUMEM_ENABLE: '1'
SGLANG_MOONCAKE_CUSTOM_MEM_POOL: 'True'
SGLANG_USE_MESSAGE_QUEUE_BROADCASTER: '0'
SGLANG_DISABLE_TP_MEMORY_INBALANCE_CHECK: '1'
SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK: '512'
SGLANG_MOE_NVFP4_DISPATCH: '1'
SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE: '1'
SGLANG_ENABLE_THINKING: '1'
SGLANG_CLIP_MAX_NEW_TOKENS_ESTIMATION: '8'
SGLANG_REASONING_EFFORT: max
PIP_BREAK_SYSTEM_PACKAGES: '1'
UCX_TLS: ib,cuda_copy,cuda_ipc,sm,self
NVSHMEM_REMOTE_TRANSPORT: none
SGLANG_DG_CACHE_DIR: /deepgemm_cache
FLASHINFER_WORKSPACE_BASE: /flashinfer_cache
SGLANG_SIMULATE_ACC_LEN: '2.5'
SGLANG_SIMULATE_ACC_METHOD: match-expected
SGLANG_SIMULATE_ACC_TOKEN_MODE: real-draft-token
sglang_config:
prefill:
served-model-name: nvidia/GLM-5.2-NVFP4
trust-remote-code: true
quantization: modelopt_fp4
kv-cache-dtype: fp8_e4m3
disaggregation-mode: prefill
disaggregation-transfer-backend: nixl
tensor-parallel-size: 4
data-parallel-size: 4
expert-parallel-size: 4
enable-dp-attention: true
enable-dp-lm-head: true
load-balance-method: total_tokens
chunked-prefill-size: 65536
max-prefill-tokens: 16384
max-running-requests: 16
cuda-graph-max-bs: 16
disable-cuda-graph: true
nsa-prefill-backend: trtllm
nsa-decode-backend: trtllm
moe-runner-backend: flashinfer_cutlass
fp4-gemm-backend: flashinfer_cutlass
enable-flashinfer-allreduce-fusion: true
weight-loader-prefetch-checkpoints: true
model-loader-extra-config: '{"enable_multithread_load": true}'
mem-fraction-static: 0.8
context-length: 1048576
enable-hierarchical-cache: true
hicache-write-policy: write_back
hicache-size: 135
hicache-io-backend: direct
speculative-algorithm: EAGLE
speculative-num-steps: 1
speculative-eagle-topk: 1
speculative-num-draft-tokens: 2
enable-metrics: true
enable-cache-report: true
decode:
served-model-name: nvidia/GLM-5.2-NVFP4
trust-remote-code: true
quantization: modelopt_fp4
kv-cache-dtype: fp8_e4m3
disaggregation-mode: decode
disaggregation-transfer-backend: nixl
disable-radix-cache: true
speculative-algorithm: EAGLE
speculative-num-steps: 2
speculative-eagle-topk: 1
speculative-num-draft-tokens: 3
tensor-parallel-size: 4
data-parallel-size: 1
expert-parallel-size: 1
enable-dp-attention: false
enable-dp-lm-head: false
max-running-requests: 16
cuda-graph-max-bs: 16
chunked-prefill-size: 64
context-length: 1048576
nsa-prefill-backend: trtllm
nsa-decode-backend: trtllm
moe-runner-backend: flashinfer_trtllm
fp4-gemm-backend: flashinfer_cutlass
skip-tokenizer-init: true
stream-interval: 30
enable-flashinfer-allreduce-fusion: true
weight-loader-prefetch-checkpoints: true
model-loader-extra-config: '{"enable_multithread_load": true}'
mem-fraction-static: 0.9
disaggregation-decode-extra-slots: 0
enable-metrics: true
enable-cache-report: true
health_check:
max_attempts: 1440
interval_seconds: 10
benchmark:
type: custom
command: bash /infmax-workspace/benchmarks/multi_node/agentic_srt.sh
env:
INFMAX_CONTAINER_WORKSPACE: /infmax-workspace
RESULT_DIR: /logs/agentic
PORT: "8000"
IS_MULTINODE: "true"
AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: "true"
AIPERF_DATASET_MMAP_CACHE_DIR: /aiperf_mmap_cache
HF_HUB_CACHE: /hf_hub_cache
infra:
etcd_nats_dedicated_node: false
nats_max_payload_mb: 64
sbatch_directives:
mem: '0'
cpus-per-task: '144'
srun_options:
mem: '0'
container-remap-root: ''
# ################# agentx #################
zip_override_mtp_agentx_hightpt:
name: [agentx-2p1d_dep16-c128, agentx-2p1d_dep16-c192]
backend:
sglang_config:
decode:
cuda-graph-max-bs: 144
data-parallel-size: 16
deepep-config: /configs/deepep_config.json
deepep-mode: low_latency
enable-dp-attention: true
enable-dp-lm-head: true
ep-dispatch-algorithm: static
ep-num-redundant-experts: 0
expert-parallel-size: 16
max-running-requests: 144
mem-fraction-static: 0.85
moe-a2a-backend: deepep
moe-dense-tp-size: 1
moe-runner-backend: flashinfer_cutedsl
speculative-moe-a2a-backend: deepep
speculative-moe-runner-backend: deep_gemm
tensor-parallel-size: 16
prefill:
data-parallel-size: 8
expert-parallel-size: 8
max-prefill-tokens: 8192
tensor-parallel-size: 8
resources:
decode_nodes: 4
decode_workers: 1
gpus_per_decode: 16
gpus_per_prefill: 8
prefill_nodes: 4
prefill_workers: 2
zip_override_mtp_agentx_lowlat:
name: [agentx-1p2d_tp4-c48, agentx-1p4d_tp4-c48, agentx-1p6d_tp4-c45]
resources:
decode_nodes: [2, 4, 6]
decode_workers: [2, 4, 6]
gpus_per_decode: 4
gpus_per_prefill: 4
prefill_nodes: 1
prefill_workers: 1
Loading
Loading