Custom-P multigrid builds above 16 ranks: UW3's copy of PETSc's nested interpolator, agreed build failures (#902, #886) by lmoresi · Pull Request #937 · underworldcode/underworld3

lmoresi · 2026-11-04T06:47:27Z

Custom-P multigrid failed at np 32 and 52 on kaiju (#902). Each level's node coordinates come from DMCreateInterpolation between two DMs on one plex, which is PETSc's DMPlexComputeInterpolatorNested.

Its preallocation loop runs only on a rank with local rows or columns. Its fill loop runs on every rank that holds cells. The guard came in with PETSc b4ec2976 (v3.24.0) and is still on main.
A shared point belongs to the highest rank that holds it, so a rank can hold cells and own none of their nodes. That rank skips the preallocation but still sends its cells' entries.
The owner then raises "New nonzero ... caused a malloc" in MatAssemblyEnd, and the other ranks wait in the assembly.
On kaiju the 124-cell coarse level left such a rank at np 32 and 52.

A failed build was then retried on the failing rank alone, so the ranks met different collectives and deadlocked (#886).

The change

UW_DMCreateInterpolationWithinPlex (petsc_compat.h) is UW3's copy of the nested interpolator for the same-plex finite-element case.
- It follows PETSc v3.25.0's plexfem.c line by line, without the guard. It adds an FE-only check and a one-plex check, and it is collective. PETSc's BSD-2 notice is in the header.
- It is used by Mesh._get_coords_for_basis (every custom-P level), project_to_degree and boundary_flux._trace_interior_coords.
UW_DMPlexCreateCoordinateSpaceWithinPlex installs a mesh's degree >= 2 coordinate space without PETSc's projection and fills it through the copy. Mesh(degree=2) failed during setup on such a rank. Isoperiodic coordinates keep PETSc's route.
custom_mg.build_transfers: a rank-local exception decides a collective retry #886. CollectiveBuildError is a build failure whose verdict every rank holds:
- the barycentric builder's agreed verdict;
- the allreduced checks in custom_mg and in fault_mg's provenance transfer.
In parallel only that error is retried or degraded; any other exception is raised. The fault split's replay agrees on any exception, and verdicts run on a communicator of their own (uw.mpi.verdict_comm).

Measured on kaiju (1 Nov): the production faulted annulus

Five steps from TAf's checkpoint 20, on one node, with a frictionless split fault, custom-P FMG and rotated walls.

build	np	custom FMG	step 21 vrms	slip_max	Newton
development `fc67b65`	16	built	634.58	2071.47	35
development `fc67b65`	32	failed (error code 98), fell back; no step within 20 minutes
this branch (`96d39f7`)	16	built	634.58	2071.47	35
this branch (`96d39f7`)	52	built	634.62	2070.76	35

At np 52 the velocity iterations of each increment match np 16 to within one.
The branch also built and completed at np 32. Those runs are not compared, because Rotated free-slip with a given (analytic) normal drops owned wall points from the constraint set at some rank counts (np 24/32/40/52; clean at np 16) #935's unconstrained wall nodes changed their time step.
The branch tip before the merge, 57d8778, ran as part of kaiju's build C at np 16, 32 and 52, with the same answers.

Tests

tests/parallel/test_1107 (level 2, tier B, np >= 2). It uses two shell partitions: rank 0 on an interior cell (owning no vertex or edge), and on a boundary cell (owning only its wall edge). It covers node coordinates at degree 1 to 3, projection between degrees, a degree-2 mesh, and a custom-P FMG solve over the interior partition (4 iterations). With PETSc's route put back, these fail at np 2.
tests/parallel/test_1108 covers a builder failing on one rank and a check failing on every rank. Both hang on development.
tests/parallel/test_1089 gains the split replay over an IndexError, and a provenance check's CollectiveBuildError.
The copy against PETSc's createInterpolation: max |A - B| = 0, with the same nonzeros, in 114 of 114 cases at np 1 and 2.

The merge with development

09c7f48 merges development through the #932-#934 FAC stack (60dbf0e), with no conflict.

The branch's own diff against development is the one the reviews saw. Development's FAC changes to custom_mg now sit beside it.
Serially after the merge, on macOS: test_1134 (2 passed) and test_1019 (15 passed).
The parallel tests run in CI's MPI phase at np 2. Kaiju's numbers above predate the merge.

Closes #902. Closes #886. Left open in this territory:

Verdicts are in the review comment.

Underworld development team with AI support from Claude Code

PETSc's DMPlexComputeInterpolatorNested skips its preallocation pass on a rank with no local rows and no local columns but fills from that rank's cells all the same. A shared point belongs to the highest rank holding it, so a rank whose cells sit wholly inside higher ranks' regions owns nothing, and its entries reach owners that never preallocated them: the owner raises "New nonzero caused a malloc" inside MatAssemblyEnd, and the rank that did not raise waits in the assembly's collective. On kaiju this stopped custom-P multigrid at np 32 and 52 (Mesh._get_coords_for_basis on the 124-cell coarsest level of the faulted annulus). The test forces the case with a shell partition: rank 0 keeps one interior cell of a 4 x 4 grid and owns none of its vertices and edges. It pins the degree 1 and 2 node coordinates and the projection between degrees (project_to_degree, which interpolates between a sub-DM and a clone of one plex the same way). On development at np 2 the degree-2 coordinates raise "New nonzero at (88,32)" on rank 1 and the projection fails after it; a thread timeout turns a hang into a failure. Underworld development team with AI support from Claude Code

…rank (#886) build_transfers retries a failed barycentric build with the RBF builder and degrades to the default preconditioner after that, each rank on its own verdict: a rank whose builder raised retried alone while the others went on into the transfer's MPIAIJ assembly. The test makes the barycentric builder raise on rank 1 only, on a single-field fmg Poisson at np 2. On the current branch it times out with rank 0 inside _assemble_node_transfer; the contract is the RBF rescue on every rank, multigrid installed, the solve converged. Underworld development team with AI support from Claude Code

…902) Mesh._get_coords_for_basis and function.project_to_degree interpolated between two DMs on one plex with createInterpolation, which takes PETSc's DMPlexComputeInterpolatorNested. That function skips its preallocation on a rank with no local rows and no local columns but fills from that rank's cells, so on a rank that holds cells and owns none of their nodes the owners raise inside MatAssemblyEnd and the other ranks wait in its collective. custom-P multigrid reads every level's node coordinates, and this stopped kaiju's faulted annulus at np 32 and 52. The element matrix PETSc applies is the same for every cell (it comes from the reference element alone). utilities.closure_interpolation builds it on one reference cell, on one rank, applies it to each local cell's closure, and hands each node its owner's value through a local-to-global and back. That covers every field whose edge and face points carry at most one node (any discontinuous field, continuous degree 1 and 2). A field with more nodes per edge, ordered by the edge's orientation, keeps PETSc's matrix, and on a partition where that matrix cannot be assembled it raises on every rank instead of hanging. Against PETSc's matrix at np 1 and 4, on triangles, quads, tets, hexes, a degree-2 annulus and a spherical shell, continuous and discontinuous degree 1 and 2: identical bits for continuous P1/P2 on simplices and Q1, round-off (at most 1.6e-15) elsewhere, 2-7x faster. test_1107 passes at np 2 and 4. The remaining interpolations of this kind are marked TODO(BUG): the degree >= 3 trace coordinates of boundary_flux and PETSc's projection of a degree >= 2 coordinate field in Mesh setup. Underworld development team with AI support from Claude Code

In parallel build_transfers caught any exception from the hierarchy build on each rank alone, and that rank retried with the RBF builder (or degraded to the default preconditioner) while the others went on into the transfer's assembly: the ranks met in different collectives and the run hung. The node-transfer builders are rank-local work, so their verdict is agreed right after them (_agreed_node_transfer, used by both parallel builds): if the builder raised on any rank, TransferBuildError is raised on every rank, with each failing rank's message. In parallel build_transfers retries or degrades only on that error; any other exception is one rank's, the others are inside the build's collectives, and it is raised where it happens. Serial keeps the retry for every failure. test_1108 (the barycentric builder raising on rank 1 only) passes at np 2 and 4: every rank takes the RBF rescue and the solve converges on geometric multigrid. Underworld development team with AI support from Claude Code

…rpolator (#902) The cell-by-cell Python route (37ac408) is replaced by a copy of PETSc's DMPlexComputeInterpolatorNested (v3.25.0, BSD-2-Clause), trimmed to the same-plex finite-element case, with its preallocation loop run over the same cells as its fill loop: UW_DMCreateInterpolationWithinPlex in cython/petsc_compat.h, petsc_dm_create_interpolation_within_plex in petsc_discretisation. PETSc has guarded only the preallocation by the local matrix sizes since b4ec2976050 (v3.24.0; still so in v3.25.6, v3.26.0 and main), so a rank that holds cells but owns no rows and no columns fills entries nobody preallocated. Mesh._get_coords_for_basis, function.project_to_degree and boundary_flux._trace_interior_coords call it in place of createInterpolation. It works with any PETSc build, covers degree 3 (two nodes per edge, which the Python route could not), and reproduces PETSc's matrix exactly: max |A - B| 0 and the same nonzeros on triangles, quads, tets, hexes, a degree-2 annulus and a spherical shell, continuous and discontinuous degree 1 to 3, at np 1 and 4, so the node coordinates are bit-identical to development. Every DM section is created on every rank before anything rank-dependent. The Python route created them only on ranks holding cells, so an empty rank went on to the next collective while the others waited (the review found test_0774/0781 at np 2 and test_0779/1103 at np 4 hanging); all four pass. test_1107 now also pins degree 3 on the forced partition, and its partition check is one verdict for every rank. PETSc's own projection of a degree >= 2 coordinate field in Mesh setup stays marked TODO(BUG). Underworld development team with AI support from Claude Code

f4b9128 let only a builder failure be retried or degraded in parallel, so the checks in CustomMGHierarchy.build whose verdict every rank already holds (the finest map against the operator, DOFs divisible by nodes, one component count, all on allreduced sizes) now crashed a parallel solve that development had degraded on every rank together, as serial still does (found in review). They raise CollectiveBuildError, which TransferBuildError becomes: an error raised on every rank together, so build_transfers retries or degrades on it in parallel; a rank-local exception is still raised where it happens. test_1108 gains the case: the finest-map check made to fail on every rank degrades every rank to the default preconditioner and the solve converges (np 2 and 4). Underworld development team with AI support from Claude Code

…degree (#905) A discontinuous source projected to a continuous target sums the adjacent cells at each shared node. It was already on development and is outside #902, so this branch only marks the place. Underworld development team with AI support from Claude Code

…#902) The coarse level is the shell-partitioned grid on which rank 0 holds one cell and owns none of its DOFs, under the grid's refinement. The build reads each level's node coordinates, which is where kaiju's np 32/52 runs failed. With PETSc's createInterpolation routed back in, the test fails inside CustomMGHierarchy.build with "New nonzero at (88,32) caused a malloc" at np 2. On the branch it installs multigrid and agrees with GAMG to 1e-6, at np 2 and 4. Underworld development team with AI support from Claude Code

…play agrees on any exception Review round 2 found two gaps in the narrowed parallel catch of build_transfers. - provenance_transfer_parallel's two checks on allreduced counts ("a fine node belongs to no cell", "fine node(s) lie in no cell") raised a plain RuntimeError on every rank. In parallel only CollectiveBuildError is retried or degraded, so the solve crashed where development degraded it. They raise CollectiveBuildError now. - split_tail_levels caught only ValueError, RuntimeError and NotImplementedError before its collective verdict, so an IndexError on one rank after the split's last collective escaped the verdict. It catches any exception now. test_1089 is parametrised over IndexError, which failed on the previous build ("DID NOT WARN", then a hang) and passes at np 2 and 3. rotated-freeslip.md says which failures degrade in parallel and which are raised. Underworld development team with AI support from Claude Code

…; PETSc's notice, a one-plex check, sharper tests Review round 2. - Mesh(degree >= 2) on a rank that holds cells but owns none of their nodes failed during mesh setup. createCoordinateSpace projects the coordinates through PETSc's DMCreateInterpolation between the old and the new coordinate DM. UW_DMPlexCreateCoordinateSpaceWithinPlex installs the space without projecting, then copies the coordinates (same dual space, as PETSc does) or interpolates them with UW_DMCreateInterpolationWithinPlex. Coordinates injected into a higher coordinate dimension are not projected, as in PETSc. Periodic coordinates keep PETSc's own projection, which is local for isoperiodic ones. The TODO(BUG) at that call is gone. - The copy of PETSc's interpolator carries PETSc's full BSD-2 notice: the copyright line, the conditions and the disclaimer. - The copy refuses, on every rank, two DMs that are not on one plex. PETSc only dispatches there for that case. A pair on two plexes would otherwise give a wrong matrix without an error. - test_1107 runs every case on two partitions. - Rank 0 holds an interior cell, owning no vertex or edge: PETSc's P2 matrices fail there. - Rank 0 holds a boundary cell, owning its wall edge and no vertex: PETSc's P1 matrices fail there too, since the wall edge's vertex pair is in no other cell. - A degree-2 mesh on each partition checks that the vertices stay on the grid and each edge coordinate is its midpoint. - The FMG case asserts its 4 iterations, the same at np 2, 3 and 4 (GAMG takes 12). Coarse nodes displaced by 0.3 of a cell take 5. - It reads .array, not .data. - On PETSc's route, run one test at a time at np 2, these fail: the interior P2 coordinates, the projection, the degree-2 mesh and the FMG case; and the boundary P1 coordinates and projection. Degree 3 passes there, since rank 0 owns its cell's interior node, so those cases check the copy's node placement. On the branch all 13 pass at np 2, 3 and 4. - The test_1107 and test_1108 docstrings no longer claim the thread timeout stops a hang inside PETSc. petsc4py holds the GIL there, and the MPI supervisor's silence limit is what ends it. Underworld development team with AI support from Claude Code

…w runs (#902) field_projection's module docstring, the XDMF guide's account of vertex projection, and the base-level coordinate repair in _DMLevelView, which said a non-FE coordinate field gives a zero matrix. UW3's copy refuses such a field instead. Underworld development team with AI support from Claude Code

…re is not retried with RBF Review round 3. - The split replay catches any exception before its verdict. A rank that raised before one of the split's own allgathers then entered the verdict's allgather on COMM_WORLD, and paired with the others' allgather inside the split. It took their data as the failure reason ("(None, 76)" for an IndexError) and left with a different hierarchy. Verdicts now go through uw.mpi.verdict_comm, a Dup of COMM_WORLD used for nothing else, so they meet only verdicts. The raise before a collective still hangs, at a known place: the raising rank in the verdict, the others in the split. That part is #874. The two comments that said it hangs "inside the split" now say where. - A CollectiveBuildError that is not a builder failure (the provenance checks, the operator sizes) is no longer retried with the RBF builder. The provenance transfer uses no builder, so the retry repeated the same failure after a full rebuild, and recorded a builder fallback that never happened. CollectiveBuildError carries builder_failure, set by the agreed builder verdict only. The degrade record names only the builders tried. - test_1089 gains the provenance degrade case. One fine node on rank 1 is made to belong to no cell, and every rank degrades to GAMG with one provenance call and no builder fallback, to the serial slip. On the previous build it failed with two calls ([2, 2] != [1, 1]). test_1089, test_1107 and test_1108 pass at np 2 and 3. Underworld development team with AI support from Claude Code

…eshes; no space, no change Review round 3. - In PETSc v3.25 only isoperiodic coordinates (DMPlexGetIsoperiodicFaceSF) are projected differently: locally, with no matrix. maxCell/L periodicity projects its non-localized coordinates through DMCreateInterpolation and MatMult, the step UW's copy replaces. Such meshes were sent to PETSc's route, which still fails on a rank owning no nodes. Now only the isoperiodic case keeps it. - For a cell type PETSc builds no Lagrange space for (tensor prisms), the coordinate DM is not replaced, and the copy called PetscFEGetDualSpace on the old PetscContainer field. Now an unchanged coordinate DM leaves the coordinates alone, as PETSc does. No UW constructor builds such cells. Underworld development team with AI support from Claude Code

…fix/custom-mg-many-ranks Clean merge; 99 commits behind before it. Underworld development team with AI support from Claude Code

lmoresi · 2026-11-04T06:47:35Z

Pre-PR adversarial review

Three rounds by independent reviewers, each followed by fixes. The records follow as written at the time. After round 3 the branch merged development (09c7f48), see the end.

Adversarial review, round 1 (bugfix/custom-mg-many-ranks at `f4b9128`)

Reviewed by a separate agent against the checklist in docs/developer/guides/adversarial-review.md. Probes ran at np <= 4.

Findings and what was done

The cell-by-cell route (37ac408) hung on a rank holding no cells. getLocalSection() on a fresh clone is created lazily and collectively, and it was called only inside the branch for ranks with cells. test_0774 and test_0781 hung at np 2, and test_0779 and test_1103 at np 4.
Resolution: the route was replaced by UW3's copy of PETSc's nested interpolator (89037a1). It creates every section on every rank before anything rank-dependent. All four tests pass.
custom_mg.build_transfers: a rank-local exception decides a collective retry #886: checks whose verdict every rank already holds now crashed a parallel solve that development had degraded. These are the finest map against the operator, DOFs divisible by nodes, and one component count. The narrowed catch in f4b9128 let them propagate.
Resolution: they raise CollectiveBuildError (e255858). test_1108 gains the case, and the solve degrades on every rank at np 2 and 4.
project_to_degree from a discontinuous to a continuous field sums the adjacent cells (2-7x at P1 vertices). This was already on development, and the exact copy keeps it.
Resolution: not fixed in this branch; filed as project_to_degree from a discontinuous field to a continuous one sums the adjacent cells #905. The XDMF writer is not affected: it projects a discontinuous field to DG0.
test_1107's partition check asserted on rank 0 only, before a collective. A broken premise would have shown up as a timeout.
Resolution: it is one verdict for every rank now (89037a1).
Direct mpi4py in the Python helper. Moot: the helper was removed.

Issue verdicts

custom_mg at np 32 and 52: the nested interpolation fails on preallocation, the FMG build falls back and the ranks deadlock #902: fixed in the branch for the interpolation. It stays open until kaiju's np 32/52 run of the faulted annulus matches np 16, which is the issue's own "Done when". Still live and marked TODO(BUG): PETSc's own projection of a degree >= 2 coordinate field in Mesh setup.
custom_mg.build_transfers: a rank-local exception decides a collective retry #886: partly fixed. The build_transfers verdicts are agreed (test_1108). Still live: the fault_mg replay catches only (ValueError, RuntimeError, NotImplementedError) before its collective verdict.
Fault-split multigrid: rank-local raises before collectives in the cell-located transfer and the line cut's orientation vote #874: still live. The cell-located transfer and the line cut are untouched.
Fault-split multigrid: a coarse level with no cells on a rank fails in custom_mg and the cell-located transfer #873, cell_size() diverges on a rank that owns no cells (np=8, tiny mesh) — and that mesh already hangs in construction on development #698: no verdict here. They need np >= 4 / np 8 probes on development.
Fault-split integration tests: cross-partition transfer and empty-rank cases not exercised in CI #888, 39 parallel test files (tests/parallel/ptest_*.py) never run in CI, and the coverage check cannot see them #900: untouched.

Checked and sound

UW3's copy against PETSc's createInterpolation at np 1 and 4: max |A - B| = 0 and the same nonzeros on triangles, quads, tets, hexes, a degree-2 annulus and a spherical shell, for continuous and discontinuous degree 1-3. The coordinates are bit-identical to development.
localToGlobal(INSERT) copies owned points only (PETSc's DMLocalToGlobalBegin).
_agreed_node_transfer is reached on every rank in both parallel builds.
test_1108 is not vacuous: on development it hangs, with rank 0 in _assemble_node_transfer.
CI reach: scripts/check_test_coverage.py reports all 382 test files reachable, and the MPI phase enumerates tests/parallel/test_*.py.
The serial retry tests (test_1020, test_1022) are unchanged: 18 passed.

Adversarial review, round 2 (bugfix/custom-mg-many-ranks at `96d39f7`, plus the then-uncommitted FMG case of test_1107)

Reviewed by a separate agent against docs/developer/guides/adversarial-review.md. Probes ran at np <= 3.

Findings and what was done

An agreed failure crashed a parallel solve that development degraded. fault_mg.provenance_transfer_parallel raised its two allreduced checks as a plain RuntimeError on every rank. In parallel, build_transfers retries or degrades only on CollectiveBuildError.
Resolution (7b39d2c): both checks raise CollectiveBuildError. Any other exception in a parallel build is still raised, not retried. It may be one rank's while the others are inside the build's collectives, and a rank cannot tell which. rotated-freeslip.md now says so.
Mesh(degree >= 2) failed during setup on a rank that owns nothing. createCoordinateSpace projects the coordinates through PETSc's DMCreateInterpolation between the old and the new coordinate DM.
Resolution (57c0de9): UW_DMPlexCreateCoordinateSpaceWithinPlex installs the space without projecting, then copies the coordinates (same dual space, as PETSc does) or interpolates them with UW3's copy. Coordinates injected into a higher coordinate dimension are not projected, as in PETSc. Periodic coordinates keep PETSc's own projection. test_1107 gains a degree-2 mesh on each partition.
PETSc's BSD-2 notice was incomplete.
Resolution (57c0de9): the copyright line, conditions and disclaimer are in the header.
The FMG case's agreement check could not catch a wrong transfer.
Resolution (57c0de9): it asserts its 4 iterations, the same at np 2, 3 and 4 (GAMG takes 12). Coarse nodes displaced by 0.3 of a cell take 5; displacements of 0.1 and below still take 4. The transfer's input, each level's node coordinates, is pinned exactly by the coordinate cases. The case reads .array.
The interior-cell partition cannot fail at P1.
Resolution (57c0de9): every case also runs with rank 0 on a boundary cell, where it owns its wall edge and no vertex. One test at a time at np 2 on PETSc's route, these fail:
- interior P2 coordinates, the projection, the degree-2 mesh, the FMG case;
- boundary P1 coordinates, the projection.
Degree 3 passes there, since rank 0 owns its cell's interior node; those cases check the copy's node placement.
The docstrings overstated the thread timeout. petsc4py holds the GIL inside PETSc collectives.
Resolution (57c0de9): they say the MPI supervisor's silence limit ends such a hang.
The C copy had no one-plex check.
Resolution (57c0de9): two DMs whose cone arrays differ are refused on every rank (clones share their plex's cone array).
Stale text in field_projection.py, create_xdmf.md and _DMLevelView.
Resolution (b148115).

Also fixed on the reviewer's #886 verdict (7b39d2c): fault_mg.split_tail_levels caught only three exception types before its collective verdict, so an IndexError on one rank escaped. It catches any exception now. test_1089's replay case is parametrised over IndexError: on the previous build it failed with "DID NOT WARN" and then hung; now it passes at np 2 and 3.

Issue verdicts

custom_mg at np 32 and 52: the nested interpolation fails on preallocation, the FMG build falls back and the ranks deadlock #902: fixed in the branch for the interpolation, the node coordinates, the projections and the mesh coordinate space. It stays open until kaiju's np 32/52 run of the faulted annulus matches np 16 (its "Done when").
custom_mg.build_transfers: a rank-local exception decides a collective retry #886: fixed in the branch for build_transfers and the split replay. A rank-local raise before the split's own collectives still hangs there; that part is Fault-split multigrid: rank-local raises before collectives in the cell-located transfer and the line cut's orientation vote #874.
Fault-split multigrid: rank-local raises before collectives in the cell-located transfer and the line cut's orientation vote #874: still live. cell_located_transfer_parallel raises rank-locally ("a fine node belongs to no cell", the owner < 0 checks), and the line cut's orientation vote is untouched.
Fault-split multigrid: a coarse level with no cells on a rank fails in custom_mg and the cell-located transfer #873: still live in fault_mg (the same owner < 0 raise as Fault-split multigrid: rank-local raises before collectives in the cell-located transfer and the line cut's orientation vote #874). The custom_mg part looks fixed on development by aed17e2, by inspection only, with no probe run.
project_to_degree from a discontinuous field to a continuous one sums the adjacent cells #905: still live, marked in project_to_degree. The copy reproduces PETSc's discontinuous-to-continuous matrix exactly.
Fault-split integration tests: cross-partition transfer and empty-rank cases not exercised in CI #888: still live. test_1103 needs np 4, and CI's MPI phase runs np 2.
custom_mg: last_builder_stats is module-global, and outside_hull counts nodes beyond a curved boundary's chords #885: still live. The RBF retry, reachable in parallel now, does not write last_builder_stats.
cell_size() diverges on a rank that owns no cells (np=8, tiny mesh) — and that mesh already hangs in construction on development #698: not adjudicated (needs np 8).

Checked and sound

UW3's copy against PETSc's createInterpolation: max |A - B| = 0, with the same nonzeros and sizes, in 114 of 114 cases at np 1 and 2. The cases cover triangles, tets, quads, hexes, the degree-2 annulus and a spherical manifold, with coordinates P0-P3, continuous and discontinuous, and projections between them.
The copy follows PETSc v3.25.0 plexfem.c line by line, apart from the FE-only check, the one-plex check and the removed guard. Rows come from the fine DM's sections and columns from the coarse DM's; sizes from the constrained storage sizes; the matrix type from the coarse DM, with MATIS mapped to AIJ.
Every section is created before anything rank-dependent.
The serial build_transfers is unchanged: any exception is retried or degraded.
Every collective in the parallel build is reached on a global decision.
test_1107 and test_1108 carry level_2, tier_b and mpi(min_size=2). CI's MPI phase enumerates tests/parallel/test_*.py.

Adversarial review, round 3 (bugfix/custom-mg-many-ranks at `b148115`)

Reviewed by a separate agent against docs/developer/guides/adversarial-review.md. Probes ran at np <= 3. The baseline for mesh construction was development plus an unrelated FAC change.

Findings and what was done

The split replay's verdict could pair with an allgather inside the split. The replay catches any exception before its verdict, and the verdict was an allgather on COMM_WORLD, like the split's own.
- Probe at np 2: an IndexError on rank 1 before one of the split's allgathers.
- Rank 1 took rank 0's data, (None, 76), as the failure reason, and left with a different hierarchy. Rank 0 waited in the verdict.
Resolution (2a9fa3e): verdicts go through uw.mpi.verdict_comm, a Dup of COMM_WORLD used for nothing else, so they meet only verdicts. A raise before a collective still hangs, at a known place: the raising rank in the verdict, the others in the split. That part is Fault-split multigrid: rank-local raises before collectives in the cell-located transfer and the line cut's orientation vote #874. Two comments that said it hangs "inside the split" now say where.
No test reached the provenance checks' CollectiveBuildError.
Resolution (2a9fa3e): test_1089 gains the case. One fine node on rank 1 is made to belong to no cell, and every rank degrades to GAMG, to the serial slip. On the previous build the test failed (two provenance calls).
A provenance failure was retried with the RBF builder. The provenance transfer uses no builder, so the retry repeated the failure after a full rebuild, and recorded a builder fallback that never happened.
Resolution (2a9fa3e): CollectiveBuildError.builder_failure is set only by the agreed builder verdict, and only that case is retried. The degrade record names only the builders tried.
The coordinate space sent maxCell/L periodic meshes to PETSc's projection. In PETSc v3.25 only isoperiodic coordinates are projected differently; maxCell/L periodicity uses the same DMCreateInterpolation, so it still failed on a rank owning no nodes.
Resolution (57d8778): only the isoperiodic case keeps PETSc's route.
Tensor prisms: an FE call on a non-FE coordinate field. PETSc installs no Lagrange space for them.
Resolution (57d8778): an unchanged coordinate DM leaves the coordinates alone, as PETSc does. No UW constructor builds these cells.
Pre-existing: Mesh(degree=2) hangs when a rank holds no cells. The FVM cell-size call fails on degree-2 coordinates ("Edge has 6 coordinates != 4", printed even at np 1), and its error is ignored on the ranks with cells.
Resolution: not fixed in this branch, filed as Mesh(degree=2): the FVM cell-size call fails and its error is ignored, so a rank with no cells hangs the others #908.
Pre-existing: a collective behind a rank-local guard in Mesh.remesh (_get_coords_for_basis under old_var.data.size > 0).
Resolution: not fixed in this branch, filed as Mesh.remesh: a collective call sits behind a rank-local guard in the variable transfer #909.
Pre-existing: degree-2 basis coordinates vary run to run at np 3, by up to 1e-15. Two cells insert the same entry with last-bit differences, and stash arrival order picks one. Development shows the same.
Resolution: noted, no action.

Issue verdicts

custom_mg at np 32 and 52: the nested interpolation fails on preallocation, the FMG build falls back and the ranks deadlock #902: fixed in the branch for a rank that holds cells but owns none of their nodes: the interpolation, node coordinates, projections and the mesh coordinate space, periodic maxCell/L meshes now included. Still live:
- degree >= 2 with a rank holding no cells, Mesh(degree=2): the FVM cell-size call fails and its error is ignored, so a rank with no cells hangs the others #908;
- kaiju's np 32/52 run, the issue's "Done when".
custom_mg.build_transfers: a rank-local exception decides a collective retry #886: fixed in the branch for build_transfers and for a replay failure after the split's last collective. A raise before a collective is Fault-split multigrid: rank-local raises before collectives in the cell-located transfer and the line cut's orientation vote #874.
Fault-split multigrid: rank-local raises before collectives in the cell-located transfer and the line cut's orientation vote #874: still live. fault_mg raises rank-locally on owner < 0. A raise before a collective in the replay now waits at a known place instead of pairing with the wrong exchange.
Fault-split multigrid: a coarse level with no cells on a rank fails in custom_mg and the cell-located transfer #873: the custom_mg part is already fixed on development. A probe at np 3, with a coarse level holding 0 cells on rank 2, gives pc mg in 4 iterations on both builds. The fault_mg part is still live.
project_to_degree from a discontinuous field to a continuous one sums the adjacent cells #905: still live, marked in project_to_degree.
Fault-split integration tests: cross-partition transfer and empty-rank cases not exercised in CI #888: still live. test_1103 needs np 4, and CI's MPI phase runs np 2.
custom_mg: last_builder_stats is module-global, and outside_hull counts nodes beyond a curved boundary's chords #885: still live. RBF does not write last_builder_stats.
cell_size() diverges on a rank that owns no cells (np=8, tiny mesh) — and that mesh already hangs in construction on development #698: the cell_size() part is already fixed on development: a probe at np 3 with one rank holding 0 cells returns on every rank on both builds. The np 8 construction hang is not adjudicated.

Checked and sound

Coordinates are bit-identical to development across 16 constructions:
- meshes: 2-D and 3-D simplex boxes, a refined box, a quad box, the annulus, the spherical shell, the cubed sphere and the cdim-3 spherical manifold;
- each at degree 1 and 2, before and after a deform;
- at np 1, 256 of 256 arrays.
- At np 3, the degree-2 basis arrays differ only at the development-against-itself 1e-15 of item 8.
- adapt(), extract_region and extract_surface at np 2: 160 of 160 arrays.
UW_DMPlexCreateCoordinateSpaceWithinPlex matches PETSc v3.25's DMPlexCreateCoordinateSpace and DMSetCoordinateDisc:
- the same FE, dual-space test, and copy or MatMult;
- balanced references on every path;
- every branch decided on values equal on all ranks.
The one-plex check is equivalent to PETSc's dmc->data == dmf->data for every caller.
The FMG case's == 4 has a wide margin: at iteration 3 the residual is 13x above rtol, at iteration 4 45x below.
test_1107, test_1108 and test_1089 pass at np 2 and 3. All 382 files are reachable in CI.

The merge with development (`09c7f48`)

A clean merge through the The FAC cover gate takes a split fault level's patch up to 98 % of its rows (#918) #932-The finest level's FAC blocks are cut from every rank's rows (#925) #934 FAC stack (60dbf0e); the branch's own diff against development is the one reviewed above.
Serially after the merge: test_1134 2 passed, test_1019 15 passed. The parallel tests run in CI's MPI phase.
Since round 3, kaiju closed the issue's "Done when" (1 Nov, on 96d39f7 and on 57d8778 with Rotated free-slip annulus: the FMG coarse level is a dense SVD on one rank, so more ranks make the solve slower (caps kaiju at np 16) #901): at np 52 the faulted annulus matches np 16 (vrms 634.62 against 634.58, slip_max 2070.76 against 2071.47, Newton 35 and 35); development fails the build at np 32.

Verdict update: #902 fixed by this branch; #886 fixed for build_transfers and the split replay (a raise before a collective is #874); the others as in round 3.

Underworld development team with AI support from Claude Code

Copilot

🟡 Changes recommended

The compatibility call uses the wrong signature for supported PETSc 3.21–3.23 versions, disabling projection and potentially breaking extension builds even for nonperiodic meshes.

1 open issue

Use the correct periodic coordinate signature for PETSc versions below 3.24 · New

What changed in this PR

This PR addresses custom-P multigrid failures at higher MPI rank counts by fixing interpolation preallocation and coordinating build-failure recovery across ranks.

Changes:

Adds a PETSc-derived interpolator that handles ranks holding cells but owning none of their nodes.
Restricts parallel retries and fallback to agreed failures, with a separate communicator for fault error verdicts.
Adds parallel regression tests and updates interpolation and recovery documentation.

File	Description
`tests/parallel/test_1108_custom_mg_builder_failure_collective_parallel.py`	Tests coordinated builder retries and fallback.
`tests/parallel/test_1107_interpolation_rank_owning_nothing_parallel.py`	Tests interpolation, coordinate setup, and multigrid on targeted partitions.
`tests/parallel/test_1089_fault_split_multigrid_parallel.py`	Extends provenance-failure and split-replay tests.
`src/underworld3/utilities/fault_mg.py`	Marks agreed failures and broadens split-replay exception handling.
`src/underworld3/utilities/fault_contact.py`	Moves error verdicts to a dedicated communicator.
`src/underworld3/utilities/custom_mg.py`	Coordinates builder failures and limits parallel recovery.
`src/underworld3/utilities/boundary_flux.py`	Uses the replacement interpolator for interior coordinates.
`src/underworld3/mpi.py`	Creates the error-verdict communicator.
`src/underworld3/function/field_projection.py`	Routes degree projection through the replacement interpolator.
`src/underworld3/discretisation/discretisation_mesh.py`	Updates coordinate-space setup and basis-coordinate interpolation.
`src/underworld3/cython/petsc_extras.pxi`	Declares the new C helpers.
`src/underworld3/cython/petsc_discretisation.pyx`	Exposes coordinate-space and interpolation wrappers.
`src/underworld3/cython/petsc_compat.h`	Implements interpolation and coordinate-space compatibility helpers.
`docs/developer/subsystems/rotated-freeslip.md`	Documents parallel multigrid recovery constraints.
`docs/beginner/create_xdmf.md`	Clarifies how vertex interpolation is assembled.

_{Review effort: Balanced}

+    PetscFunctionBeginUser;
+    PetscCall(DMPlexGetIsoperiodicFaceSF(dm, &nfaceSF, NULL));
+    if (nfaceSF) {
+        PetscCall(DMPlexCreateCoordinateSpace(dm, degree, PETSC_FALSE, PETSC_TRUE));


…ned-node-936 #938's branch, which this one was stacked on, is now in development; clean merge. Underworld development team with AI support from Claude Code

lmoresi added 14 commits October 31, 2026 13:31

Merge development (through the #932-#934 FAC stack, 60dbf0e) into bug…

09c7f48

…fix/custom-mg-many-ranks Clean merge; 99 commits behind before it. Underworld development team with AI support from Claude Code

Copilot AI balanced review requested due to automatic review settings November 4, 2026 06:47

Copilot started reviewing on behalf of lmoresi November 4, 2026 06:48

View session

Copilot AI reviewed Nov 4, 2026

View reviewed changes

Comment thread src/underworld3/cython/petsc_compat.h

PetscFunctionBeginUser;

PetscCall(DMPlexGetIsoperiodicFaceSF(dm, &nfaceSF, NULL));

if (nfaceSF) {

PetscCall(DMPlexCreateCoordinateSpace(dm, degree, PETSC_FALSE, PETSC_TRUE));

lmoresi merged commit c514013 into development Nov 4, 2026
9 checks passed

lmoresi mentioned this pull request Nov 5, 2026

Memory grows ~190 MB per rank per fault rebuild on development f631ce42 (rotated free slip + custom FMG); 25d62812 did not leak, and #882's regression test does not cover this path #948

Open

Navigation Menu

Repository navigation

Custom-P multigrid builds above 16 ranks: UW3's copy of PETSc's nested interpolator, agreed build failures (#902, #886) - #937

Custom-P multigrid builds above 16 ranks: UW3's copy of PETSc's nested interpolator, agreed build failures (#902, #886)#937
lmoresi merged 14 commits into
developmentfrom
bugfix/custom-mg-many-ranks

lmoresi commented Nov 4, 2026

Uh oh!

lmoresi commented Nov 4, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

lmoresi commented Nov 4, 2026

The change

Measured on kaiju (1 Nov): the production faulted annulus

Tests

The merge with development

Uh oh!

lmoresi commented Nov 4, 2026

Pre-PR adversarial review

Adversarial review, round 1 (bugfix/custom-mg-many-ranks at f4b9128)

Findings and what was done

Issue verdicts

Checked and sound

Adversarial review, round 2 (bugfix/custom-mg-many-ranks at 96d39f7, plus the then-uncommitted FMG case of test_1107)

Findings and what was done

Issue verdicts

Checked and sound

Adversarial review, round 3 (bugfix/custom-mg-many-ranks at b148115)

Findings and what was done

Issue verdicts

Checked and sound

The merge with development (09c7f48)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Adversarial review, round 1 (bugfix/custom-mg-many-ranks at `f4b9128`)

Adversarial review, round 2 (bugfix/custom-mg-many-ranks at `96d39f7`, plus the then-uncommitted FMG case of test_1107)

Adversarial review, round 3 (bugfix/custom-mg-many-ranks at `b148115`)

The merge with development (`09c7f48`)