Skip to content

adapt: snap_level_boundaries guards its allreduce on a rank-local mask (np>1 hang) #627

Description

@lmoresi

snap_level_boundaries reaches its allreduce only on ranks that own a snapped vertex, so adapt with registered bounding surfaces can hang at np>1.

In src/underworld3/discretisation/discretisation_mesh.py the inversion check
after a boundary snap is guarded by a rank-local mask:

snapped_any = numpy.zeros(arr.shape[0], dtype=bool)
for s in _snap_surfs:
    mask = sync_mask(_pinned_mask(dm, (s.label,)))
    if mask.any():
        arr[mask] = s.restore(arr[mask])
        snapped_any |= mask
dm.setCoordinatesLocal(vec)
if snapped_any.any():              # <-- rank-local
    ...
    if uw.mpi.size > 1:
        flipped = uw.mpi.comm.allreduce(flipped, op=_MPI.SUM)

snapped_any is a mask over this rank's local vertices. sync_mask makes it
agree between ranks that share a vertex, but a rank whose partition holds no
vertex on any registered surface has an all-False mask, skips the block, and
never arrives at the reduction. Its peers stay in allreduce until the job is
killed.

This is the shape in #612 and in the migrate working set: a collective behind a
predicate on local data. It is partition-dependent, so it will present as an
intermittent hang in adapt rather than a reproducible one, and it needs both
np>1 and a partition that leaves a rank clear of the snap surfaces — which is
why it has not shown up in the np=2 suite.

Fix: initialise flipped = 0 before the guard, keep the cell-orientation
loop inside it (the local work is genuinely conditional), and reach the
reduction unconditionally.

Found by a static scan of every comm.<op> call site asking whether it is
reached on all ranks; it flags 2 of 210 sites, this one and #612.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions