The finest level's FAC blocks are cut from every rank's rows (#925)

docs/developer/design/fault-patch-multigrid-2026-08.md

@@ -271,10 +271,19 @@ to permit it.)
    update the star forest consistently (the known np>=3 line-cut issue;
    the independent-pass design is the template). Along-strike block
    cuts at seams are measured free (the k-ladder).
+   > [2026-11-03] Not for a zone-keyed patch: cut at rank boundaries (a TI
+   > band beside a split fault), its blocks took the velocity solve from 4
+   > iterations serially to 118 at np 2 and past 200 at np 4. The structural
+   > patch alone, cut the same way, took 67 iterations serially and 445 at
+   > np 16 (row 5, n_rungs 320). Both are now built from every rank's rows
+   > (#925, `test_1139`).
 2. **PCASM is rank-local.** Blocks become per-rank lists with local
    indices; faultless ranks carry `nsd = 0`, and every IS-building path
    must survive empty sets (the empty-stratum `getIndices()` SEGV
    family, #589).
+   > [2026-11-03] The finest level's blocks, a keyed zone's or the structural
+   > patch's, are per-rank lists of GLOBAL rows: each block goes whole to the
+   > rank owning most of its rows and may hold other ranks' rows (#925).
 3. **Pair detection must go collective or be replaced by zone keying.**
    Rank-local coordinate matching misses pairs that straddle a boundary
    and can false-match ghosts; the zone mask is rank-local by nature and

docs/developer/subsystems/fault-split-multigrid.md

@@ -72,11 +72,14 @@ largest block, and each rank's block sizes. It is made once, with one allgather
 `CustomMGHierarchy.install` and the rotated path's two `_configure_pcmg` calls.
 Reading it makes no collective call, so any rank can read it alone. A driver that
 looped a reduction over its own rank's blocks hung at np 4, because the ranks made
-different numbers of calls. The structural patch's owned rows are disjoint across
-ranks, so their total does not depend on the partition. A zone block's rows include
-the seam halo, which two ranks can both hold, so its summed rows can exceed the serial
-count while the union of the ranks' rows equals it (`test_1133`). The record is each
-block as handed to the installer, before PCASM adds its overlap layer at set-up.
+different numbers of calls. With no zone keyed, the summed patch (owned) rows are the
+serial count at any rank count (`test_1133`), and the subdomain rows follow the
+structural halo, which can be a few rows short of serial (#896). With a zone keyed,
+its blocks and the structural block beside them hold the same rows as patch and
+subdomain, so both counts carry that halo. Each block of the finest level is held
+whole by one rank (see below, #925), and `block_rows_per_rank` shows where each sits:
+the structural LU on one rank. The record is each block as handed to the installer, before PCASM
+adds its overlap layer at set-up.
 
 ## Rotated walls on every level
 
@@ -140,8 +143,8 @@ path.
   | S 0.004, blocks of 2000 rows | 433 s | 1404 s | 2089 / 2058 s | 2248 / 2981 s |
   | S 0.015625, one block | 516 s | 766 s | 931 / 869 s | 923 / 1207 s |
 
-The size of the LU blocks is bounded by `fac_block_rows` for keyed zones; the
-structural block is one block per rank (#895).
+The size of the LU blocks is bounded by `fac_block_rows` for keyed zones. The
+structural block is not bounded: it is one LU, held by one rank in parallel (#895).
 
 ## Parallel
 
@@ -296,6 +299,28 @@ holds the band cells but not the fill around them. A level without a split keys
 patch on the masks alone. That is #670's default, measured on a placed weak plane
 when the patch replaced the level smoother (`test_1135`).
 
+In parallel the finest level's blocks are built from every rank's rows. Each mask's
+rows are gathered and segmented along strike as one band, with the two nodes of a
+split pair kept in one piece, so the zone's pieces are the same at any rank count.
+The structural rows outside every mask are one block, and with no zone keyed the
+structural patch is one block; its halo can differ from the serial one by a few rows
+where a partition boundary runs along the band (#896). Each block goes whole to the
+rank that owns most of its rows, so the structural block is one LU on one rank, as in
+serial. That rank carries the factorisation the others do not
+(`patch_summary`'s `block_rows_per_rank`); balancing it is #895's question. Built
+rank by rank, a rank's blocks held seam rows that its neighbour's blocks also held,
+and rank boundaries cut the band and the structural block. On row 5 at n_rungs 80
+the velocity solve then took 4 iterations serially, 118 at np 2 and more than 200 at
+np 4 (#925). Gathered, it takes 4 at np 1, 2 and 4 (`test_1139`). The structural
+patch alone, one block per rank, converged but took 3 iterations serially, 4 at np 2
+and 6 at np 4; gathered it is one block at any rank count (`test_1139`, `test_1133`).
+On row 5 with no zone at n_rungs 320 (30 048 rows in the block; kaiju, 2026-11), both
+take 67 iterations serially, where the code is the same. Cut per rank, the velocity
+solve took 128, 131, 316 and 445 iterations at np 2, 4, 8 and 16; as one block, 76 to
+86. The owning rank's peak memory was 512-541 MB at np 2-16; the others'
+fell from 379 MB at np 2 to 248-284 MB at np 16 (1.35 times at np 2, 1.6-2.2 times
+at np 4-16).
+
 ## Not yet done
 
 - A regression test for the coarse-level wall rotation itself (a rotated free-slip

scripts/test.sh

@@ -129,6 +129,12 @@ fi
 # Re-measure over several runs before re-assigning: runner variance here is of
 # the same order as the imbalance left.
 #
+# 2026-11-03: shard 4 had grown to 55:32 on development (run 41564997729, #930)
+# and ran past the 60-minute cap on PR #934, against 29 min for shards 2 and 3.
+# Shard 4 owns the whole --p N phase, and every new file under tests/parallel/
+# had also put its serial run in batch 4. Those serial runs (about 6 min) moved
+# to shard 3, and test_05*/06*/07* (about 4 min) to shard 2.
+#
 # The floor is test_106*/test_107*, and it is a TAIL, not a volume: 48% of its
 # 149 tests finish in the first 2 minutes, then the remainder takes 31. Since
 # the distribution is --dist loadfile, a file is atomic on a worker, so the
@@ -298,7 +304,7 @@ if [ $PARALLEL_ONLY -eq 0 ]; then
   # whole band was measured 2026-09-12 and passes; a band is not a unit of trust,
   # and holding one back for a suspicion nobody recorded meant its level_1 files
   # never ran at all.
-  batch 4 tests/test_05*py tests/test_06*py tests/test_07*py
+  batch 2 tests/test_05*py tests/test_06*py tests/test_07*py
 
   # Units system tests (0800-0899: unit-aware functions, arrays, and conversions)
   batch 3 tests/test_08*py
@@ -319,24 +325,26 @@ if [ $PARALLEL_ONLY -eq 0 ]; then
   # spherical shell at np=1). Run it here so the serial job covers that path —
   # every other serial test of the default boundary normal is on a box, where
   # flat walls make the question vacuous. It also runs in the --p N batch below.
-  batch 4 tests/parallel/test_1069_boundary_normal_parallel.py
+  batch 3 tests/parallel/test_1069_boundary_normal_parallel.py
   # The live multigrid configuration (#892) is the same check in serial and in
   # parallel, so it is one file under tests/parallel/ with no min_size mark too.
-  batch 4 tests/parallel/test_1090_mg_live_configuration_parallel.py
+  batch 3 tests/parallel/test_1090_mg_live_configuration_parallel.py
   # So is the live finest-smoother operator of the rotated Newton loop (#894).
-  batch 4 tests/parallel/test_1088_rotated_interface_operator_parallel.py
+  batch 3 tests/parallel/test_1088_rotated_interface_operator_parallel.py
   # So is the band patch, the same rows at any rank count (#896).
-  batch 4 tests/parallel/test_1106_fac_cover_partition_parallel.py
+  batch 3 tests/parallel/test_1106_fac_cover_partition_parallel.py
   # So is the static mark of a moved base finest (#919).
-  batch 4 tests/parallel/test_1132_addon_partition_static_mark_parallel.py
+  batch 3 tests/parallel/test_1132_addon_partition_static_mark_parallel.py
   # So is the FAC patch summary, the same on every rank (test_1133).
-  batch 4 tests/parallel/test_1133_fac_patch_summary_parallel.py
+  batch 3 tests/parallel/test_1133_fac_patch_summary_parallel.py
   # So is the serial reference child's environment (#922).
-  batch 4 tests/parallel/test_1138_serial_reference_child_env.py
+  batch 3 tests/parallel/test_1138_serial_reference_child_env.py
   # So is the split level's FAC patch, taken over the old cover gate (#918).
-  batch 4 tests/parallel/test_1134_fac_cover_gate_split_parallel.py
+  batch 3 tests/parallel/test_1134_fac_cover_gate_split_parallel.py
   # So is the zone-keyed patch of a split level, which keeps its structural rows (#921).
-  batch 4 tests/parallel/test_1135_fac_zone_split_structural_parallel.py
+  batch 3 tests/parallel/test_1135_fac_zone_split_structural_parallel.py
+  # So are the zone-keyed patch's blocks, the same at any rank count (#925).
+  batch 3 tests/parallel/test_1139_fac_zone_blocks_any_rank_count_parallel.py
   # test_106*/test_107* were held back pending "a triage/deselect decision". That
   # decision is made: the band is not the unit of exclusion. Of the 243 tests in
   # the deferred bands only 26 are level_3 or tier_c; 55 are level_1 AND tier_a —