[lmoresi]

In parallel each rank cut the finest level's FAC blocks from its own rows, so the blocks depended on the partition:

• A zone block took in the ghost rows of the rank's masked cells. A row on a partition seam then sat in blocks on two ranks, and basic ASM added both corrections there: 98 of 8296 patch rows at np 2 and 218 of 8298 at np 4 on test_1139's fixture.
• Rank boundaries cut the band and the structural block into pieces that the serial cut never makes.

Of the two, the cut is what stops the solve converging. With a TI zone keyed beside a split fault, the union #921 makes the default, the velocity solves ran to their 200 cap at np 2 and 4 even with every row held once. Built from the serial blocks gathered across ranks, they took 4-6 iterations.

This PR is stacked on the #921 PR (bugfix/fac-zone-structural-921).

Change

• _gathered_patch_blocks builds the finest level's blocks in parallel. Rank 0 gathers every rank's rows and cuts them as the serial branch does:
  • each mask is one band, segmented along strike at fac_block_rows, with nodes taken by coordinate so that the two nodes of a split pair stay in one piece;
  • the structural rows outside every mask are one block;
  • with no zone keyed, the structural patch and its halo are one block.
• Each block goes whole to the rank that owns most of its rows, and rank 0 scatters them. Only rank 0 segments, so no two ranks can cut differently.
• The blocks are the same at any rank count. They differ from serial in two documented ways:
  • the structural halo is _fac_patch_split_parallel's, which can differ by a few rows where a partition boundary runs along the band (FAC structural patch declined at np 2 on the linear fault fixture: 640 velocity iterations instead of 68 #896);
  • with two masks at a small cap a piece boundary can fall a node away from serial, because serial orders a split pair's two nodes separately.
• A keyed zone wins over detection in parallel too. A keyed mask that held no cell on any rank fell back to the structural split in parallel, while serial installs nothing; now both install nothing. The zone_anywhere allreduce is gone.
• The keying must be the same on every rank. CustomMGHierarchy does one allgather of each rank's mask count and raises ValueError on every rank if the counts differ; rank 0 keyed and the others not would leave rank 0 in the zone's gather and the others in the cover gate's allreduce. An empty list of masks raises too. Pass a mask on every rank, all-False where a rank holds no zone cell, or None on every rank; both docstrings say so.
• patch_summary's block_rows_per_rank shows which rank holds each block. Its docstring, fault-split-multigrid.md and the 2026-08 patch design record (dated notes where FAC zone plus a split fault's structural rows: the solve hangs at np 4 (converges in serial) #925 contradicts it) are updated.

Measured on kaiju (Linux)

The zone case: row 5 of the add-on matrix, a TI band keyed beside the split fault, at n_rungs 80. Velocity iterations per solve, and their sum:

ranks	before	after
1	4-5 (88)	the same code
2	118-200, 6 of 19 solves at the cap (3179)	4-6 (97)
4	200 on all 39 solves (7800)	4-6 (95)

At n_rungs 320 (h_b/S = 24) and np 4, the outer solve before this branch did not finish its first iteration in 20 minutes: 31 of 33 velocity solves ran to the cap. After it, the velocity solve takes 110 and 117 iterations at np 2 and 4, against 94 serially.

The structural patch alone: row 5 with no zone, n_rungs 320, 30 048 rows in the block. Cut per rank it converged, but its iterations grew with the rank count; as one block they stay near serial.

ranks	cut per rank: velocity its	one block: velocity its	one block: peak memory, owning rank / others
1	67	67	549 MB
2	128	76	512 / 379 MB
4	131	80	514 / 280-314 MB
8	316	82	519 / 254-288 MB
16	445	86	541 / 248-284 MB

At np 1 both columns run the serial branch. The per-rank column is e67aa11, this branch with the structural patch still cut per rank. It predates the stack's development merges, so the table compares iterations, not times. Cut per rank, the largest block was still 19 582-21 072 rows, so cutting saved little factorisation.

Row 3 (#918's case) does not change: its structural patch was already one block on one rank, and the builds with and without this change agree to 10 digits at np 1-16.

The one-rank LU is a known imbalance

The structural block is one unbounded LU on the rank that owns most of its rows, as in serial. That rank's peak memory is 1.6-2.2 times the others' at np 4-16. On row 3's 61k-row block the first solve takes 66 s serially and 63 s at np 8 and 16: the factorisation does not spread over the ranks. Balancing it is left to the parallel decomposition of the add-on (phase 3), under #895. #895's title still describes one LU per rank.

Tests

tests/parallel/test_1139_fac_zone_blocks_any_rank_count_parallel.py (new; level 2, tier B; np 1 in CI's serial phase, np 2 in the MPI phase) runs row 5 at n_rungs 80:

• Zone keyed on the split level: every patch row is in exactly one block over all ranks. There are two blocks, the zone's 1650 rows and the structural rows beside them, and the velocity solve takes at most 5 iterations (4 at np 1, 2 and 4).
• Zone alone on an unsplit level, cap 500: every row is held once, in ceil(rows / 500) pieces at any rank count.
• The structural patch alone: one block, whose size is the gate's count of patch and halo rows over all ranks, and at most 3 iterations (3 at np 1, 2 and 4).
• A keyed mask with no cell installs no patch.
• An empty list of masks, and a zone keyed on some ranks only, raise ValueError.

Each test except the unsplit-zone one failed first on kaiju at np 2 and 4, on the library before the matching change:

• rows held twice, 4 and 6 blocks, 123 and 200 iterations;
• the structural patch alone in 2 and 3 blocks, at 4 and 6 iterations;
• a 7980-row block from the empty zone;
• "DID NOT RAISE" for the two refusals.

The unsplit-zone test passes on the old library; it guards the cut.

• test_1133 asserts the zone strip as one block of 1510 rows and the structural patch as one block at every rank count. Its band lies on one rank at np 2 and 4, so these assertions guard the summary.
• test_1135 compares rows over all ranks, since each block now goes whole to one rank.

On kaiju at c094073, test_1139 (8 passed and 1 skipped at np 1, 9 at np 2 and 4), test_1133 (2) and test_1135 (2) pass at np 1, 2 and 4. tests/parallel at np 2 and 4 on fc58912's library shows no failure beyond the development baseline (test_1062, test_1064 at np 2, test_1078: #923).

scripts/test.sh gains a line in serial batch 4. The tip, a2e3839, merges the #921 branch (development through #927) and corrects the memory figures in the doc; the tests above pass serially there.

Closes #925. Left open: #895 (the one-rank LU), #901 (the same one-rank pattern on the rotated FMG's coarse level), and #897, #886, #874 and #873 (untouched); verdicts in the review comment.

Underworld development team with AI support from Claude Code

[lmoresi]

Adversarial review (round 1, on 32fc081), and what changed in e67aa11

We read the diff against docs/developer/guides/adversarial-review.md. Every check here ran serially; the parallel runs are kaiju's. We emulated the parallel branch over fake partitions: cell slabs along x or y at np 1, 2, 3, 4 and 7, with global rows renumbered rank by rank in random order. That gives 30 partitions per case.

Fixed in e67aa11

1. Ranks could cut different blocks. Each rank recomputed the segmentation, an SVD, from the gathered rows and kept the blocks it owned. If one rank's result differed in the last bit, rows would end up in no block or in two, with no error. Rank 0 now gathers, cuts and assigns, then scatters each rank its blocks. This also keeps the gathered rows off the other ranks.
2. Split pairs made the pieces partition-dependent. Nodes were keyed by global row and ordered by coordinate, so the tie between the two nodes of a split pair was broken by row number. A piece boundary then fell between them according to the partition. With two masks at cap 300, 18 of 29 partitions differed by 4-8 rows. Nodes are now taken by coordinate, so a split pair stays in one piece. 30 of 30 partitions give identical blocks at caps 2000, 500 and 300, for split, unsplit and two masks.
3. test_1135 compared per rank. It compared each rank's structural rows with that rank's blocks. A zone-keyed block now goes whole to one rank, so the rows are compared over all ranks.
4. test_1139 and the docs.
   • test_1139 now asserts the 500-row cap.
   • The docs no longer say the blocks are exactly the serial ones. The structural block's halo is _fac_patch_split_parallel's, which can differ by a few rows where a partition boundary runs along the band (FAC structural patch declined at np 2 on the linear fault fixture: 640 velocity iterations instead of 68 #896).
   • The 2026-08 design record carries dated notes where FAC zone plus a split fault's structural rows: the solve hangs at np 4 (converges in serial) #925 contradicts it ("seam cuts are free", "per-rank lists with local indices").
   • Test spacing nit fixed.

Kept

• One structural LU on one rank. With a zone keyed, the structural block is one LU on the rank that owns most of its rows, as in serial. We measured the alternative of leaving it cut per rank (probe 238338f, local mode): 2-4x the velocity iterations at n_rungs 80, and every solve at the 200 cap at n_rungs 320, np 4. The block's size and where it sits are FAC patch in 3-D: a split fault's structural block is one LU per rank (one block wins in 2-D; 3-D cost unmeasured) #895's question.
• The structural patch without a zone stayed per rank in round 1. Round 2 changed this; see below.
• The 1650-row count. test_1139 pins the zone's 1650 rows. The count is the same on macOS and on kaiju (Linux); a CI difference would show up as a per-host count, as in test_1134.

Attacks that found nothing

• Collectives. None sits behind a rank-dependent branch: the zone flag is an LOR allreduce and the split gate's verdict is global.
• Empty ranks. Ranks with no cells, rows or owned range, including equal ownership starts, run without error.
• Node identity. Ghost rows are resolved through l2g, and constrained components are dropped.
• Masks. Overlapping masks are deduplicated per mask, and the structural block excludes the rows of every mask.
• dtype. Rows are int64 up to IS creation.
• CI reach. test_1139 is in test.sh batch 4 and picked up by the tests/parallel enumeration.

Issues in this territory

Issue	Verdict
#925	Fixed by this branch: test_1139 at np 2 and 4, and row 5 at n_rungs 80 and 320 on kaiju.
#921	Fixed on the base branch; test_1135 now compares over all ranks.
#895	Still live. With a zone keyed, the structural block is one uncapped LU on one rank, as in serial.
#901	Still live; the same one-rank pattern applies to that structural block.
#896	Fixed in development. Its documented halo deviation is why the structural block can differ by a few rows from serial.
#918, #892, #902	Untouched by this diff.

Round 2 (on bed177b), and what changed in fc58912, ef7804e and 9b69572

The structural patch alone is gathered too (fc58912). Cut per rank, it converged but degraded with the rank count. On row 5 with no zone, n_rungs 80, the velocity solve took 70 / 96 / 143 iterations at np 1 / 2 / 4. The fix now builds it from every rank's rows as one block. test_1139 failed first on the old library: 2 blocks and 4 iterations at np 2, 3 blocks and 6 at np 4.

Finding 1, one unbounded LU on one rank: kept, on kaiju's measurements. We modelled the factor cost on one machine and expected the one-rank block to lose from np 8 up. The model's fixture refined the base along the fault, which is not one of the add-on's cases, so we dropped it and timed the cases we run. Row 5 with no zone at n_rungs 320 (30 048 rows in the block):

np	per rank: vel its	one block: vel its
1	67	67
2	128	76
4	131	80
8	316	82
16	445	86

At np 1 both run the serial branch, the same code. We compare iterations, not seconds, because the per-rank series ran before the stack's development merges.

Cut per rank, the largest block was still 19 582-21 072 rows, so the factor saved little while the iterations grew. The owning rank's peak memory was 512-541 MB at np 2-16, while the other ranks fell from 379 MB at np 2 to 248-284 MB at np 16: 1.35 times at np 2, 1.6-2.2 times at np 4-16. On row 3 (#918's case) the structural patch was already one block before this branch, and both overlays agree to 10 digits at np 1-16. The one-rank LU there takes 55-63 s at np 8 and 16 against 66 s serially. Balancing it is #895's question.

Fixed

• A keyed zone with no cell (ef7804e). In parallel a keyed mask that held no cell on any rank fell back to detecting the structural split, while serial installs nothing. The parallel branch now keys on fac_zone is not None, as the serial branch does, and one allreduce is gone. test_1139's new test failed first on the old library at np 2 and 4, installing a 7980-row block, and passes at np 1, 2 and 4.
• "Cuts as serial" overclaimed. Serial segments a split pair as two nodes and the gathered cut as one site, so with two masks at a small cap the pieces can differ from serial by a node. They are the same at every rank count. The docstring says so.
• The patch summary's wording. Only the owned rows sum to the serial count. The subdomain rows follow the structural halo, FAC structural patch declined at np 2 on the linear fault fixture: 640 velocity iterations instead of 68 #896's deviation.
• Stale doc line. "The structural block is one block per rank" now reads one unbounded LU, held by one rank in parallel (FAC patch in 3-D: a split fault's structural block is one LU per rank (one block wins in 2-D; 3-D cost unmeasured) #895). The 2026-08 design record's notes cover the structural patch as well as the zone.
• test_1139's bounds. The structural block's size must equal the gate's count of every rank's patch and halo rows, taken before the gather (9b69572). The iteration bound's comment records 3 iterations at np 1, 2 and 4.
• test_1133 counted a placeholder (bed177b). A rank holding no block records an empty [0] in block_rows_per_rank; the test counts non-empty blocks.

Verified on kaiju: test_1139 (7) and test_1133 (2) pass at np 1, 2 and 4 on ef7804e. Row 3 at r640/r1280 matches #918's iteration counts. The full tests/parallel run on fc58912 adds no failure beyond the baseline (test_1062, test_1064 at np 2, test_1078), apart from test_1133's two, fixed in bed177b.

Issues in this territory (round 2)

Issue	Verdict
#925	Fixed by this branch, for the zone and the structural patch alone.
#895	Still live. Its premise is stale: the structural block is no longer one block per rank but one unbounded LU on one rank, as in serial.
#897, #886, #874, #873	Still live; untouched.

Round 3 (on 9b69572), and what changed in c094073

We found nothing high. The five keying cases (band + split, empty + unsplit, empty + split, none + split, none + unsplit) install in parallel what serial installs, apart from the documented site-cut and halo differences. The new test_1139 assertion equals the gate's pre-gather count by construction, and it fails if the gather drops a rank's rows.

Fixed

• Keying that differs between ranks hung. The docstring promised the same keying on every rank, but nothing checked it. A caller writing fac_zone=mask if mask.any() else None would leave rank 0 in the zone's gather and the others in the cover gate's allreduce. CustomMGHierarchy now does one allgather of each rank's mask count at construction, and every rank raises ValueError if the counts differ. That replaces the allreduce ef7804e removed. Test first in 2e11a6f.
• fac_zone=[] was keyed with no zone. On a split level the structural block was (owned, sub) in parallel and (sub, sub) serially. It now raises. Test first in 2e11a6f; it fails serially on the old library ("DID NOT RAISE").
• Owned-rows wording. The owned rows equal the serial count only with no zone keyed. With a zone, both counts carry the structural halo.
• Scaling paragraph. It compares iterations only.
• Memory figures (a2e3839). The doc said the owning rank used about twice the others' memory at np 2 and np 16 alike; it now gives the measured 1.35 at np 2 and 1.6-2.2 at np 4-16.

a2e3839 merges the #921 branch (development through #927); test_1133/1134/1135/1136/1139 pass serially there.

Underworld development team with AI support from Claude Code