Skip to content

fix(colocate): derive num_gpus_per_node from actor_num_gpus_per_node#2012

Open
aoshen02 wants to merge 1 commit into
THUDM:mainfrom
aoshen02:fix/colocate-num-gpus-per-node
Open

fix(colocate): derive num_gpus_per_node from actor_num_gpus_per_node#2012
aoshen02 wants to merge 1 commit into
THUDM:mainfrom
aoshen02:fix/colocate-num-gpus-per-node

Conversation

@aoshen02
Copy link
Copy Markdown
Contributor

@aoshen02 aoshen02 commented Jun 3, 2026

Problem

In colocate mode the rollout engines share the actor's physical nodes, so the GPUs-per-physical-node always equals actor_num_gpus_per_node. But --num-gpus-per-node defaults to 8 (an 8-GPU/node assumption). On hardware with a different per-node GPU count (e.g. 4× GB200/node), that default is wrong for multi-node colocate runs.

_allocate_rollout_engine_addr_and_ports_normal does:

num_engines_per_node = max(1, args.num_gpus_per_node // _gpus_per_engine)
...
node_index = local_rank // num_engines_per_node

With num_gpus_per_node=8 but only 4 GPUs/node and rollout_num_gpus_per_engine=2, num_engines_per_node becomes 4 instead of 2, so every engine maps to node_index=0 and is handed the head node's IP. Worker-node engines then fail to bind:

OSError: [Errno 99] Cannot assign requested address
  -> server exited unexpectedly

The help text already documents the manual workaround ("If you are going to use less than 8 gpus per node under colocate mode, you should set this number"), but it's silently easy to forget and the resulting failure is opaque.

Fix

Since the value is fully determined in colocate mode, derive it automatically from actor_num_gpus_per_node:

if args.num_gpus_per_node != args.actor_num_gpus_per_node:
    logger.info("colocate: overriding num_gpus_per_node ... -> actor_num_gpus_per_node ...")
    args.num_gpus_per_node = args.actor_num_gpus_per_node

Scope / safety

  • Single-node 8-GPU runs: unaffected — actor_num_gpus_per_node == 8 == default, so the override is a no-op.
  • Non-colocate (disaggregated) runs: untouched (the override is inside the if args.colocate: block), preserving the ability to configure a rollout cluster with a different per-node GPU count than the actor.
  • Consuming/allocation logic is unchanged.

Verification

Reproduced and fixed on a 2-node, 4-GPU/node Blackwell colocate run:

  • Before: all rollout engines report the head node's IP → worker engines Errno 99.
  • After: log shows colocate: overriding num_gpus_per_node 8 -> 4; engines split correctly across head (engines 0,1) and worker (engines 2,3); bind succeeds.

In colocate mode the rollout engines share the actor's physical nodes, so
the GPUs-per-physical-node always equals actor_num_gpus_per_node.
`--num-gpus-per-node` defaults to 8 (an 8-GPU/node assumption). On hardware
with a different per-node GPU count (e.g. 4x GB200/node) that default is
wrong for *multi-node* colocate runs.

`_allocate_rollout_engine_addr_and_ports_normal` computes
`num_engines_per_node = num_gpus_per_node // rollout_num_gpus_per_engine`
and then `node_index = local_rank // num_engines_per_node`. With
num_gpus_per_node=8 but only 4 GPUs/node and rollout_num_gpus_per_engine=2,
num_engines_per_node becomes 4 instead of 2, so every engine maps to
node_index=0 and is handed the head node's IP. Worker-node engines then
fail to bind:

    OSError: [Errno 99] Cannot assign requested address
    -> vLLM/SGLang server exited unexpectedly

The help text already documents the manual workaround ("if you use less
than 8 gpus per node under colocate mode, you should set this number"),
but it is silently easy to forget and the failure is opaque. Since the
value is fully determined in colocate mode, derive it automatically.

Single-node 8-GPU runs are unaffected (actor_num_gpus_per_node == 8 ==
default, so the override is a no-op). Non-colocate (disaggregated) runs are
untouched, preserving the ability to configure a rollout cluster with a
different per-node GPU count than the actor.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant