snap_level_boundaries reaches its allreduce only on ranks that own a snapped vertex, so adapt with registered bounding surfaces can hang at np>1.
In src/underworld3/discretisation/discretisation_mesh.py the inversion check
after a boundary snap is guarded by a rank-local mask:
snapped_any = numpy.zeros(arr.shape[0], dtype=bool)
for s in _snap_surfs:
mask = sync_mask(_pinned_mask(dm, (s.label,)))
if mask.any():
arr[mask] = s.restore(arr[mask])
snapped_any |= mask
dm.setCoordinatesLocal(vec)
if snapped_any.any(): # <-- rank-local
...
if uw.mpi.size > 1:
flipped = uw.mpi.comm.allreduce(flipped, op=_MPI.SUM)
snapped_any is a mask over this rank's local vertices. sync_mask makes it
agree between ranks that share a vertex, but a rank whose partition holds no
vertex on any registered surface has an all-False mask, skips the block, and
never arrives at the reduction. Its peers stay in allreduce until the job is
killed.
This is the shape in #612 and in the migrate working set: a collective behind a
predicate on local data. It is partition-dependent, so it will present as an
intermittent hang in adapt rather than a reproducible one, and it needs both
np>1 and a partition that leaves a rank clear of the snap surfaces — which is
why it has not shown up in the np=2 suite.
Fix: initialise flipped = 0 before the guard, keep the cell-orientation
loop inside it (the local work is genuinely conditional), and reach the
reduction unconditionally.
Found by a static scan of every comm.<op> call site asking whether it is
reached on all ranks; it flags 2 of 210 sites, this one and #612.
snap_level_boundariesreaches itsallreduceonly on ranks that own a snapped vertex, soadaptwith registered bounding surfaces can hang at np>1.In
src/underworld3/discretisation/discretisation_mesh.pythe inversion checkafter a boundary snap is guarded by a rank-local mask:
snapped_anyis a mask over this rank's local vertices.sync_maskmakes itagree between ranks that share a vertex, but a rank whose partition holds no
vertex on any registered surface has an all-False mask, skips the block, and
never arrives at the reduction. Its peers stay in
allreduceuntil the job iskilled.
This is the shape in #612 and in the migrate working set: a collective behind a
predicate on local data. It is partition-dependent, so it will present as an
intermittent hang in
adaptrather than a reproducible one, and it needs bothnp>1 and a partition that leaves a rank clear of the snap surfaces — which is
why it has not shown up in the np=2 suite.
Fix: initialise
flipped = 0before the guard, keep the cell-orientationloop inside it (the local work is genuinely conditional), and reach the
reduction unconditionally.
Found by a static scan of every
comm.<op>call site asking whether it isreached on all ranks; it flags 2 of 210 sites, this one and #612.