[lmoresi]

In parallel, partition_addon moved a base's own finest level and marked the moved copy _mg_static = False. Mesh.add_fault then took that background level for an add-on level and replayed the fault onto it (split_tail_levels). On the UWTN 2026-022 fixture the replay failed with a warning and left the level welded. Serial never attempts the replay, so how the hierarchy was built depended on the rank count.

Change

• partition_addon marks the moved finest static when the mesh it partitions owns no tail, which is what makes that mesh a base. An adapt child owns a tail and stays add-on.
• The base never moves inside partition_addon, so the first level's recorded parents are already current addresses, and they are now taken as they stand. Without this, a moved base finest that becomes the base of a later partition (the one add_fault(cut=True) makes of the placed mesh) resolved its parents through the record of its own earlier move: 3076 cells outside their parent at np 2.
• _moved_wrap keeps a moved level's own mark (18e5714). On its own that commit did not fix In parallel, the add-on partition marks the base's finest level as an add-on level, so the fault is replayed onto a background level #919: the base mesh carries no mark (only its wraps do), so the moved copy stayed unmarked, and test_1132 still failed at np 2 and 4. The line stays because it gives the right mark to a moved level that does carry one.
• fault-split-multigrid.md and the parallel add-on design note say what moves when the base has no add-on levels.

Tests

tests/parallel/test_1132_addon_partition_static_mark_parallel.py (level 2, tier B; np 1 in CI's serial phase, np 2 in the MPI phase) places the 80-rung fixture without a solve. It asserts:

• no add-on level in the mesh-owned tail;
• no split tail level;
• no carry warning;
• every cell's centroid lies in the parent its record names.

parents_contain_centroids moves from test_1096 into fault_fixtures.py, so there is one copy.

Fail first (kaiju):

• development and 18e5714: AssertionError [2] at np 2;
• 7c0c62f: 3076 cells outside their parent at np 2, RuntimeError / SIGSEGV at np 4.

c7e6fa3 on kaiju (Linux, PETSc 3.25.0):

• test_1132 passes at np 1, 2 and 4, and test_1096 at np 2 and 4.
• tests/parallel at np 2 and 4, and tier A+B, show no failure beyond the development baseline.
• The HDF5 error-76 casualties pass on reruns at low load.

The validation details are in #919.

scripts/test.sh gains a line in serial batch 4 at the same place as the patch summary PR and the serial_reference PR, so the second and third to merge need a development merge first.

Closes #919. Left open in this territory: #873, #874, #886, #888 and #898 (verdicts in the review comment).

Underworld development team with AI support from Claude Code

[lmoresi]

Pre-PR adversarial review

One round on 7c0c62f (over 25d6281). Each finding was fixed in c7e6fa3, the fix the review proposed, and kaiju confirmed it.

HIGH 1: a moved base finest became the base of the next partition, and _resolve_parents read that level's stale move record.

• Marking the moved host static made _levels_of return it as base when add_fault(cut=True) partitions the placed mesh.
• _resolve_parents(base, lev) then mapped the first level's current parent addresses through the host's own earlier move directory (_addon_self_old). On a rank holding more host cells than before, the indices run past the directory.
• The result was wrong rows in the (host, finest) transfer on the mesh-owned tail, at np > 1 only. At 25d6281 the host was add-on and moved with the placed mesh, so the relation was consistent there.
• Fixed: for l == 0 the recorded parents are taken as they stand, because the base never moves inside partition_addon. test_1132 asserts parent containment. On kaiju it fails at 7c0c62f (3076 cells outside their parent at np 2; RuntimeError / SIGSEGV at np 4) and passes at c7e6fa3 at np 1, 2 and 4.

LOW 2: the partition_addon comment overclaimed and told history. Fixed: it states the constraint.

LOW 3: docs. fault-split-multigrid.md and parallel-addon-distribution-2026-10.md said no base level moves. Fixed, and the history and the overlong line are gone.

LOW 4: test_1132's np 1 claim had no CI run. Fixed: a serial batch 4 line.

LOW 5: 18e5714's _moved_wrap line is untested and alone fixes nothing. Kept, and the PR body says so. It gives the same mark to every level that can reach it.

Attacks that failed:

• rank asymmetry in the new predicates;
• a single-level base;
• adapt children with and without add-on levels;
• re-partitioning a moved adapt child;
• extruded meshes;
• CI reach at np 2.

Issue verdicts

• In parallel, the add-on partition marks the base's finest level as an add-on level, so the fault is replayed onto a background level #919: fixed by this branch.
• Fault-split multigrid: rank-local raises before collectives in the cell-located transfer and the line cut's orientation vote #874: still live. The rank-local if (owner < 0).any(): raise in cell_located_transfer_parallel is untouched.
• Fault-split multigrid: a coarse level with no cells on a rank fails in custom_mg and the cell-located transfer #873: still live in part. nb = cn.shape[1] in fault_mg still reads an empty rank's array.
• custom_mg.build_transfers: a rank-local exception decides a collective retry #886: still live. There is still a per-rank except Exception retry in custom_mg; the branch removes one trigger, not the mechanism.
• Fault-split integration tests: cross-partition transfer and empty-rank cases not exercised in CI #888: still live. This branch adds no cross-partition-builder or empty-rank coverage.
• remesh() leaves the multigrid hierarchy on the pre-remesh mesh: adapt() then raises or takes the old mesh as a level #898: still live, untouched.
• Three parallel tests fail on the HPC container at np 2/4 and pass in CI: test_1062 (2 %), test_1078 (2.3e-3), test_1064 spherical3d topography; their serial baselines come from another host #923: outside this territory. The kaiju failure set is the same at 25d6281 and on the branch.

Underworld development team with AI support from Claude Code

[Copilot]

🟡 Changes recommended

The new test’s tier_b marker is applied at module scope, contrary to the style charter’s requirement for tier markers on individual tests.

1 open issue

• Add required tier_b marker to the test · New

What changed in this PR

This PR addresses #919 by keeping a base mesh’s moved finest level classified as static background, preventing rank-dependent fault replay in the multigrid hierarchy.

Changes:

• Preserve static marks and use current parent addresses when partitioning above an already-moved base.
• Add serial/MPI regression coverage and share the parent-containment helper.
• Document which background levels move during partitioning.

File	Description
tests/​parallel/​test_1132_addon_partition_static_mark_parallel.py	Adds checks for static classification, absent fault replay, and parent containment.
tests/​parallel/​test_1096_addon_partition_parallel.py	Reuses the shared parent-containment helper.
tests/​parallel/​fault_fixtures.py	Houses the extracted parent-containment helper.
src/​underworld3/​utilities/​addon_partition.py	Corrects static marking and first-level parent-address handling.
scripts/​test.sh	Includes the new regression in serial batch 4.
docs/​developer/​subsystems/​fault-split-multigrid.md	Explains moved background levels and subsequent partition behavior.
docs/​developer/​design/​parallel-addon-distribution-2026-10.md	Clarifies the base-finest movement exception.

_{Review effort: Balanced}