[jmarrero-bot]

This adds a tmt test that stacks a directory sysext (over /usr) and a directory confext (over /etc) on a bootc host whose /var is a separate partition. The partition is empty when the host first boots from it, and it holds the extensions.

The test merges and unmerges each extension from a shell. It then replaces a merged extension with a new version and checks that refresh shows the new one. Finally it checks that systemd-sysext.service and systemd-confext.service merge them at boot from the separate /var, and that no unit has failed once that boot has settled. Each failure starts with the component that broke: [layout], [bootc], [systemd-sysext], [systemd-confext] or [test harness].

The change touches only the test file and the regenerated tmt lists. Booting a disk installed inside the VM isn't possible here, and root fills the rest of the VM's disk. So, like the XBOOTLDR test, the test shrinks the ESP, which holds about 8M of its 512M, to 128M. It puts the /var partition in the space that frees and mounts it from /etc/fstab. Only tmt's plan directory and scripts, taken from tmt's environment, are copied onto it.

The test runs on CentOS Stream 9 and 10 in image mode, on the ostree backend only. That is one VM, with three boots, in one matrix cell per distribution. systemd-confext only exists from systemd 254 on, so CentOS Stream 9 (systemd 252) tests the sysext and logs a SKIP for the confext. On newer systemd, a missing systemd-confext fails the test.

Finding: on CentOS Stream 9, merging relabels /usr (systemd/systemd#34387, fixed in 257 by systemd/systemd#35132):

• After a merge from a shell, /usr is user_tmp_t, and sshd refuses all new logins with chroot("/usr/share/empty.sshd"): Permission denied.
• After a merge by the service, /usr is tmpfs_t.

On systemd older than 257, the test reports exactly the label each way of merging gives as a KNOWN ISSUE, and fails on any other label change. It unmerges before tmt has to log in again. CentOS Stream 10's systemd 257 keeps the /usr and /etc labels, so the check is strict there. tmt has no result that warns without failing the plan, so the known issue only shows in the test's output. A RHEL backport would keep systemd at 252, so a TODO next to the version gate says to drop the exception then.

Finding: a merged confext keeps /etc read-only for the whole boot. On CentOS Stream 10, bootc's own bootc-publish-rhsm-facts.service then fails with Writing etc/rhsm/facts/bootc.facts: Read-only file system. The test reports that one unit as a KNOWN ISSUE until bootc skips it on a read-only /etc, and fails on any other unit that fails.

Not covered: bootc upgrade or switch with extensions merged.

Testing was on a 16-core devspace (bcvk 0.19.0, tmt 1.80.0). Images were built like CI's ostree cells, at this tree. Logs are in https://gist.github.com/jmarrero-bot/ebbba1edfa9d7a805f304a789afec21b.

• just test-tmt-nobuild extensions-separate-var passed on centos-9 (sysext, with the label KNOWN ISSUE lines) and on centos-10 (sysext and confext, with the rhsm KNOWN ISSUE).
• These deliberately broken variants each failed with a tagged message:
  • sysext extension-release, on both distros: [systemd-sysext] ... is not visible under /usr.
  • confext extension-release, on c10: [systemd-confext] ... is not visible under /etc.
  • replacing the extension without a refresh, on both: [systemd-sysext] ... holds '... v1', expected '... v2'.
  • systemd-confext treated as missing, on c10: [systemd-confext] systemd 257 should have systemd-confext.
  • a unit that fails 60s into the final boot, on c10: [systemd-confext] late-fail-demo.service failed ....
• just validate passed.

Part of jmarrero-forge/tracker#88. The docs half is a separate PR.

Generated-by: https://github.com/jmarrero-bot#llms

---

Review draft in jmarrero-forge, not upstream yet. This section is removed when the PR is opened upstream.

• Upstream: bootc-dev/bootc, base main
• Board item: PVTI_lADOFBIZ7c4BlZPrzhH73ic
• Fork CI: off; the devspace testing described above is this PR's CI, and upstream CI runs once it is opened there

To review:

• Approve, or comment /promote on a line of its own, to open it upstream, ready for review. Either covers only the commits pushed so far.
• If upstream requires DCO, approving also signs off: promote adds Signed-off-by: Joseph Marrero Corchado <jmarrero@redhat.com> to the commits lacking it (the bot's and yours; anyone else's only if you ask), with you as committer.
• Add a /draft line (in the same comment or before) to open it upstream as a draft (/ready undoes that).
• Close to drop it.
• Edit the title and description freely: they become the upstream PR's. Review comments are squashed into the commits they concern, with a reply here.

[jmarrero-bot]

Review guide for head 56ec81f5cd: where to look closely and what is safe to skim. It is advice from the bot's reviewer and doesn't replace reading the diff; the review app walks it.

Adds a centos-9-only tmt test (plan 70) that gives the bcvk VM a separate, initially empty /var partition and merges, unmerges, refreshes and boot-merges a directory sysext from it, with failures prefixed [layout], [bootc] or [systemd-sysext]. The xtask commit adds a disk_size extra field that grows the VM disk with virsh blockresize once SSH answers, because bcvk ignores --disk-size (bcvk#416) and bootc-generic-growpart.service (Before=basic.target) would take space added earlier. Risk sits in three places: the in-guest repartition and the copy of only tmt's dirs onto the new /var (depends on tmt's guest layout across tmt-reboot), the KNOWN ISSUE tolerance of the systemd 252 /usr relabel (systemd#34387), which as written can never fail in CI because CI only runs systemd 252, and the distro-only gating. The fmf files are generated from the .nu header and the rest of the xtask change is plumbing.

Hotspots

1. look-closely · logic — crates/xtask/src/tmt.rs:799-819 (386c7e4): Sound only because growpart is ordered Before=basic.target, so it is done once sshd answers; worth saying so in the comment. disk_size is an absolute size: if bcvk's default disk ever exceeds it, blockresize would shrink the disk. Plain virsh also bypasses bcvk's connect URI and retry wrapper.
2. look-closely · logic — tmt/tests/booted/test-sysext-separate-var.nu:5-9 (56ec81f): Gated on distro == centos-9 only, but the test needs the disk space only xtask adds. Under running_env=packit with distro=centos-9 the plan is enabled (checked with tmt plan ls); adding 'and running_env == image_mode' keeps Testing Farm out regardless of its distro naming.
3. note · logic — tmt/tests/booted/test-sysext-separate-var.nu:172-187 (56ec81f): Assumes root is the last partition on a plain disk (true for bcvk's GPT layout). sfdisk moves the backup GPT header on write; partx --update is what makes the new node appear, since the BLKRRPART reread fails as busy.
4. look-closely · logic — tmt/tests/booted/test-sysext-separate-var.nu:189-204 (56ec81f): Copies a snapshot of /var/tmp/tmt and /var/lib/tmt before tmt-reboot runs, so anything tmt writes there afterwards (the reboot request) stays on the old /var. This works with tmt 1.80 but is coupled to tmt's guest layout and reboot flow, and a tmt bump could break it.
5. look-closely · logic — tmt/tests/booted/test-sysext-separate-var.nu:126-139 (56ec81f): On systemd < 257, any change of the /usr label is only printed, and CI runs only 252, so this check can never fail there. Consider tolerating only the two known types (user_tmp_t, tmpfs_t) and linking a RHEL 9 backport bug for systemd#35132. Strict vs tolerant is the operator's call.
6. note · error-handling — tmt/tests/booted/test-sysext-separate-var.nu:52-57 (56ec81f): Unmerge-before-fail covers fail_in/run_in failures only. A nu runtime error while /usr is user_tmp_t (e.g. an unexpected JSON shape) leaves sshd refusing new logins, which can hide the error behind an SSH failure unless tmt reuses its multiplexed connection.
7. note · test-gap — tmt/tests/booted/test-sysext-separate-var.nu:13-15 (56ec81f): Says SELinux relabeling is out of scope, but check_usr_label asserts the /usr label. Say that what is out of scope is labeling of extension content. The extension is a directory, not a .raw image, so loop/dissect paths are not covered.

Safe to skim

• tmt/tests/tests.fmf:304-313: generated by cargo xtask update-generated from the .nu header
• tmt/plans/integration.fmf:462-475: generated by cargo xtask update-generated from the .nu header
• crates/xtask/src/tmt.rs:1143-1150: plumbing the new field through TestDef
• crates/xtask/src/tmt.rs:1351-1376: parsing the field from the test header, same pattern as the other extra fields
• crates/xtask/src/tmt.rs:1529-1541: emitting extra-disk_size into the generated plan
• crates/xtask/src/tmt.rs:1760-1777: unit test for the plan field

[jmarrero-bot]

@jmarrero I need two decisions from you. I went with the recommended option for each:

1. Where it runs: it runs in the default PR matrix's centos-9 ostree cell, where it adds about 3-5 min to one cell. If the split from bootc CI: map the test matrix, a minimal per-PR set, and path-based test selection tracker#85 lands, it moves to postsubmit with the rest of c9. The alternative is a scheduled-only job.
2. The /usr relabel on c9 (systemd#34387): the test tolerates it as a KNOWN ISSUE on systemd < 257, but only for the two labels seen (user_tmp_t, tmpfs_t), so the cell stays green. The alternative is a strict failure, which keeps the c9 cell red until RHEL 9 backports sysext: Set SELinux context for hierarchies and workdir systemd/systemd#35132. If you want pressure there, a RHEL 9 systemd backport request is the lever. I haven't filed one.

Related, for the docs PR #21: on c9 today, a systemd-sysext merge from a shell breaks new SSH logins.

[jmarrero-bot]

Review guide for head 862321366a: where to look closely and what is safe to skim. It is advice from the bot's reviewer and doesn't replace reading the diff; the review app walks it.

Adds a centos-9, image-mode-only tmt test (plan 70) that gives the bcvk VM a separate, initially empty /var partition and merges, unmerges, refreshes and boot-merges a directory sysext from it, with failures prefixed [layout], [bootc] or [systemd-sysext]. The xtask commit adds a disk_size extra field. After SSH answers, xtask grows the VM disk with virsh blockresize, and only grows it: it compares against virsh domblkinfo and fails if the disk is already that big. Waiting for SSH is required because bcvk ignores --disk-size (bcvk#416) and bootc-generic-growpart.service (Before=basic.target) would take space added any earlier. The remaining risk is in the test: copying only tmt's own dirs onto the new /var depends on tmt 1.80's guest layout, now documented. The KNOWN ISSUE tolerance of the systemd 252 /usr relabel (systemd#34387) is now limited to the two observed SELinux types; whether to be strict instead is the operator's decision (see the PR comment). Unmerge-on-failure moved to a try/catch around main. The fmf files are generated from the .nu header, and the rest of the xtask change is plumbing.

Hotspots

1. note · logic — crates/xtask/src/tmt.rs:204-238 (77000be): Sizes are binary (K/M/G/T = 2^10..2^40), like virsh's own, and are passed to blockresize as exact bytes with a B suffix. The function refuses to shrink. It parses the Capacity line of domblkinfo's default byte output, so a virsh output change would surface as a clear error, not a wrong size.
2. note · logic — crates/xtask/src/tmt.rs:834-847 (77000be): Correct only while growpart stays ordered before basic.target (and therefore before sshd). Plain virsh uses the default libvirt URI, the same one xtask's bcvk calls rely on, but it skips bcvk's session-daemon retry wrapper.
3. look-closely · logic — tmt/tests/booted/test-sysext-separate-var.nu:195-210 (8623213): Copies a snapshot of /var/tmp/tmt and /var/lib/tmt before tmt-reboot runs, so whatever tmt writes there afterwards stays on the old /var. The header now documents this dependency on tmt 1.80's guest layout and reboot flow, but a tmt bump could still break it.
4. look-closely · logic — tmt/tests/booted/test-sysext-separate-var.nu:131-145 (8623213): Policy decision for the operator: on systemd < 257, user_tmp_t and tmpfs_t on /usr are only printed as KNOWN ISSUE, so the c9 cell stays green while a real sysext bug (sshd lock-out after a shell merge) persists. Any other label, or any change on systemd >= 257, fails.
5. note · error-handling — tmt/tests/booted/test-sysext-separate-var.nu:271-286 (8623213): The catch now unmerges after any error, including whatever tmt-reboot's interruption raises, which is harmless just before a reboot. Re-raising with only $err.msg drops the original location, so a nu runtime error (as opposed to a fail_in) is reported without saying where it happened.
6. note · logic — tmt/tests/booted/test-sysext-separate-var.nu:178-193 (8623213): Assumes root is the last partition on a plain disk, which is true for bcvk's GPT layout. sfdisk relocates the backup GPT header on write; the kernel's own reread fails as busy, and partx --update is what makes the new node appear.

Safe to skim

• tmt/tests/tests.fmf:304-313: generated by cargo xtask update-generated from the .nu header
• tmt/plans/integration.fmf:462-475: generated by cargo xtask update-generated from the .nu header
• crates/xtask/src/tmt.rs:1171-1178: plumbing the new field through TestDef
• crates/xtask/src/tmt.rs:1379-1406: parsing and validating the field from the test header, same pattern as the other extra fields
• crates/xtask/src/tmt.rs:1559-1571: emitting extra-disk_size into the generated plan
• crates/xtask/src/tmt.rs:1790-1825: unit tests: table-driven parse_size and the plan field

[jmarrero-bot]

Second-opinion review at 8623213: minor fixes. The code is sound and it meets the CI half of tracker#88. The main open point is how the CS9 /usr relabel gets handled. That is a policy decision, not a bug.

1. KNOWN ISSUE has no exit plan (test-sysext-separate-var.nu:52-57,131-145). On CS9, systemd-sysext merge from a shell locks sshd out, and the test passes with a log-only line. The < 257 gate is right (systemd#35132 is in the v257 milestone), but on CS9 it is always true, since a RHEL backport would keep version 252. So nothing will ever say when the allowance can go, or when a backport regresses. Suggest filing a CentOS Stream/RHEL request to backport #35132 and linking it next to the constant. Also consider making the known issue visible in tmt's result, not only in output.txt. The docs PR shouldn't claim plain sysext works on CS9 without mentioning this.
2. The allowlist is per label, not per merge path (:131-145). If a service merge came out user_tmp_t, it would be reported as KNOWN. sshd would then fail at boot, and the failure would show up as tmt not reconnecting instead of a [systemd-sysext] line. Fix: pass the expected bug type to check_usr_label, user_tmp_t after a shell merge or refresh and tmpfs_t after the service.
3. bcvk#416 is described wrongly (crates/xtask/src/tmt.rs:34-37, commit message, PR body). That issue is about the image cache ignoring --disk-size. The reason that holds either way is that root fills the disk and growpart grows it at boot. Reword.
4. Attribution gaps in boot 0 (:195-207). If tmt's paths move, rsync fails under [layout]. Taking the paths from $env.TMT_SCRIPTS_DIR (which xtask sets) and the tmt run dir would make that clearer. Without nofail in the fstab line, a /var mount failure drops into emergency mode and surfaces as a reboot timeout. Check whether nofail gets you to check_layout on ostree.
5. Nits: the catch's systemd-sysext unmerge throws away its result, so print stderr on failure (:283). The shrink-refusal message could include the requested size. Consider skip_if_composefs: true, since this is only exercised on ostree (c9 composefs runs only readonly and image-upgrade-reboot).

Checked and fine: blockresize runs after SSH, on the per-VM qcow2 overlay rather than bcvk's cached base disk, with a correct B suffix. parse_size has overflow checks and data-driven tests, and generation-time validation is syntax only, which is fine. The partitioning works: the log shows sfdisk relocating the backup GPT, and partx and a label-based fstab do the rest. Copying only tmt's dirs holds up because root's SSH key comes from bcvk credentials, not /var/roothome, and the gist shows all 3 boots reconnecting. The unmerge-before-reboot ordering is right. The regenerated fmf entries match the generator's pattern, and validate (update-generated --check) passed in the gist. Scoping is right: c9 has exactly one ostree cell running integration, and running_env == image_mode keeps it out of Packit, where disk_size wouldn't apply. The CI cost is one VM and three boots in that cell. Commits pass bot-git check, and rustfmt --check is clean. No devspace used; I relied on the gist logs.

[jmarrero-bot]

Review guide for head 862321366a: where to look closely and what is safe to skim. It is advice from the bot's reviewer and doesn't replace reading the diff; the review app walks it.

Adds tmt test 70: on a CentOS Stream 9 ostree host, boot 0 grows the VM disk (new xtask disk_size field, virsh blockresize after SSH is up), appends a /var partition after root and mounts it from /etc/fstab with only tmt's dirs copied over; boot 1 merges/unmerges/refreshes a directory sysext from the new /var and enables systemd-sysext.service; boot 2 checks the service merged it at boot. The main risk is policy rather than code: systemd 252 on CS9 relabels /usr on merge (systemd#34387), and a shell merge then locks out sshd. The test logs that as KNOWN ISSUE and passes, and on CS9 the < 257 gate is always true (a RHEL backport keeps version 252), so nothing will flag when the allowance can go. The label allowlist is also per-label rather than per merge path. The xtask change is small and correct; it resizes the per-VM overlay, not bcvk's cached base disk. Generated fmf entries follow the generator's pattern and validate --check passed in the linked logs.

Hotspots

1. look-closely · logic — tmt/tests/booted/test-sysext-separate-var.nu:131-145 (8623213): Either bug label is accepted at every call site, so a service merge that came out user_tmp_t (which would lock sshd out at boot) is logged as KNOWN and shows up later as tmt failing to reconnect. Pass the expected label per merge path. On CS9 the version gate never trips, so this allowance is effectively permanent there.
2. note · test-gap — tmt/tests/booted/test-sysext-separate-var.nu:52-57 (8623213): A KNOWN ISSUE only appears in the log and the tmt result stays pass. Nothing links a RHEL/CentOS Stream backport request for systemd#35132, so there is no exit plan for the allowance.
3. look-closely · logic — tmt/tests/booted/test-sysext-separate-var.nu:195-207 (8623213): This depends on tmt keeping its files in /var/tmp/tmt and /var/lib/tmt. xtask sets TMT_SCRIPTS_DIR itself, so the paths could come from the environment. If tmt moves them, rsync fails and the error is tagged [layout] when it is really a harness problem. The fstab line has no nofail, so a /var mount failure ends in emergency mode and shows up as a reboot timeout, not [layout].
4. note · error-handling — tmt/tests/booted/test-sysext-separate-var.nu:279-285 (8623213): The unmerge-before-fail call throws away its own result. If the unmerge fails, tmt's failed login afterwards is the only symptom, so print stderr when it exits non-zero.
5. note · api — crates/xtask/src/tmt.rs:34-37 (77000be): bcvk#416 is about the image cache ignoring --disk-size, not about --disk-size failing to size the installed disk. The reason that holds even with bootc-dev#416 fixed is that root fills the disk and growpart grows it at boot. Reword here, in the commit message and in the PR body.
6. note · logic — crates/xtask/src/tmt.rs:219-238 (77000be): This is correct: shrink and equal sizes are refused, B means bytes for virsh, and it runs after SSH, so growpart has already finished. The bail message could include the requested size.

Safe to skim

• tmt/plans/integration.fmf: generated from the test header
• tmt/tests/tests.fmf: generated from the test header
• crates/xtask/src/tmt.rs:1790-1825: table-driven unit tests

[jmarrero]

why do we need the changes to crates/xtask/src/tmt.rs? Is this 100% needed to add this test ? Anyway to avoid changing anything other than the test files we are adding?

[jmarrero]

Also we should also be testing config extensions

[jmarrero]

Let's also test both c10s and c9s...

[jmarrero-bot]

@jmarrero All three are done in 0c04f37. It's now one commit that touches only the test file and the regenerated tmt lists.

• xtask: I dropped the change. Nothing in bcvk, tmt or the test metadata gives a test free disk space: root fills the disk, and bootc-generic-growpart grows root at boot. The test now does what test-composefs-xbootldr.nu does. It shrinks the ESP, which uses about 8M of its 512M, to 128M, and puts the separate /var partition in the space that frees.
• confext: a directory confext in /var/lib/confexts now goes through the same steps as the sysext: merge, unmerge and refresh from a shell, then systemd-confext.service at boot. Failures are tagged [systemd-confext]. On c10 it passes, and /etc keeps etc_t. systemd-confext only exists from systemd 254, so on c9 (systemd 252) the test logs a SKIP for it.
• c9s and c10s: it runs in one ostree cell per distribution. On c10 (systemd 257) the /usr label check is strict and passes. The c9 KNOWN ISSUE is now tied to how the extension was merged: user_tmp_t after a merge from a shell, tmpfs_t after the service.

Both distros passed, and the broken sysext and broken confext runs failed with the tagged messages. just validate passed too. Logs: https://gist.github.com/jmarrero-bot/ebbba1edfa9d7a805f304a789afec21b

[jmarrero-bot]

Review guide for head 0c04f37b5c: where to look closely and what is safe to skim. It is advice from the bot's reviewer and doesn't replace reading the diff; the review app walks it.

One commit that adds tmt plan 70, gated to centos-9 and centos-10 in image mode on the ostree backend. The test shrinks the VM's ESP from 512M to 128M (the same technique as test-composefs-xbootldr.nu), puts an empty XFS /var partition in the freed space and mounts it from /etc/fstab, copying only tmt's plan directory and scripts onto it. It then merges, unmerges and refreshes a directory sysext (and, on systemd 254 and later, a directory confext) from a shell, and has the services merge both at boot. The c9 /usr relabel from systemd#34387 counts as a KNOWN ISSUE only for the label each merge path is known to give. Main risk: a merged confext makes /etc read-only from before sysinit.target, and on c10 bootc's own bootc-publish-rhsm-facts.service then fails with EROFS and the system boots degraded. The test does not look for failed units, so it passes anyway. Second risk: the ESP rewrite, which keeps PARTUUID and the vfat volume ID, and which the UEFI reboots exercise. The fmf files are generated.

Hotspots

1. risky · test-gap — tmt/tests/booted/test-extensions-separate-var.nu:416-435 (0c04f37): Boot 2 never checks for failed units. On c10 the confext makes /etc read-only and bootc-publish-rhsm-facts.service fails ('Writing etc/rhsm/facts/bootc.facts: Read-only file system'), so the system is degraded, yet the test passes. Fail on systemctl --failed, tagging bootc-* units as [bootc].
2. look-closely · logic — tmt/tests/booted/test-extensions-separate-var.nu:404-413 (0c04f37): systemd-confext.service is Before=sysinit.target and systemd-tmpfiles-setup.service, so /etc is read-only for the whole of boot 2, and on c10 the units are enabled by preset. Only boot-time writers are exercised. A bootc upgrade or switch (the /etc 3-way merge and ostree-finalize-staged) with a confext merged is not covered.
3. look-closely · logic — tmt/tests/booted/test-extensions-separate-var.nu:270-337 (0c04f37): Rewrites the ESP while the system is booted. sfdisk -N keeps the partition GUID (checked), mkfs.vfat -i keeps the volume ID, and the files are copied back. The VMs boot with uefi-insecure firmware, so boots 1 and 2 really do boot through the rewritten ESP. An interruption between the umount and the copy-back would leave the VM unbootable, which shows up as a reboot timeout.
4. note · logic — tmt/tests/booted/test-extensions-separate-var.nu:159-182 (0c04f37): Policy: on systemd < 257 the label from each merge path (user_tmp_t after a shell merge, tmpfs_t after the service) is only printed as KNOWN ISSUE, and any other label fails. A RHEL backport would keep the version at 252, so the TODO is the only exit plan.
5. note · logic — tmt/tests/booted/test-extensions-separate-var.nu:237-258 (0c04f37): Copies the common prefix of TMT_TREE and TMT_PLAN_DATA (the plan directory) plus TMT_SCRIPTS_DIR. That works with tmt 1.80 on both distros, but it still assumes tmt needs nothing under /var after the copy.
6. note · error-handling — tmt/tests/booted/test-extensions-separate-var.nu:437-456 (0c04f37): The catch unmerges and reports a failed unmerge. Re-raising with only $err.msg drops the original location, so a nu runtime error (as opposed to a tagged fail_in) is reported without saying where it happened.
7. note · logic — tmt/tests/booted/test-extensions-separate-var.nu:1-20 (0c04f37): The gate was checked with tmt plan ls: the plan is enabled for centos-9 and centos-10 in image_mode and disabled under packit and on fedora. fixme_skip_if_composefs works because xtask reads BOOTC_variant=composefs, but here it expresses a scope choice, not a known bug.

Safe to skim

• tmt/tests/tests.fmf:304-313: generated by cargo xtask update-generated from the .nu header
• tmt/plans/integration.fmf:462-475: generated by cargo xtask update-generated from the .nu header

[jmarrero-bot]

@jmarrero One more finding, and a choice for you. Head is now e8e50d6.

On c10, a merged confext keeps /etc read-only for the whole boot, and bootc's own bootc-publish-rhsm-facts.service then fails (Writing etc/rhsm/facts/bootc.facts: Read-only file system). The test now fails on any unit that fails in the boot where the extensions are merged. For now it reports this one unit as a KNOWN ISSUE, so the c10 cell stays green.

I'd recommend a separate small bootc PR that skips the facts write on a read-only /etc (ConditionPathIsReadWrite=/etc on the unit, or handling EROFS in crates/lib/src/rhsm.rs), and then dropping the exception. The alternative is a strict failure now, which keeps c10 red until that PR lands. Tell me which you want.

[jmarrero-bot]

Review guide for head e8e50d6138: where to look closely and what is safe to skim. It is advice from the bot's reviewer and doesn't replace reading the diff; the review app walks it.

One commit that adds tmt plan 70, gated to centos-9 and centos-10 in image mode on the ostree backend. The test shrinks the VM's ESP from 512M to 128M (the same technique as test-composefs-xbootldr.nu), puts an empty XFS /var partition in the freed space and mounts it from /etc/fstab, copying only tmt's plan directory and scripts onto it. It then merges, unmerges and refreshes a directory sysext (and, on systemd 254 and later, a directory confext) from a shell, and has the services merge both at boot. In that final boot it now also fails on any failed unit, with two exceptions that are printed as KNOWN ISSUE instead. On c9 that is the /usr relabel from systemd#34387, for the label each merge path is known to give. On c10 it is bootc-publish-rhsm-facts.service, which fails because the confext keeps /etc read-only, and only when a confext is merged. Both exceptions are operator decisions with TODOs. The ESP rewrite keeps PARTUUID and the vfat volume ID, and the UEFI reboots exercise it. The fmf files are generated.

Hotspots

1. look-closely · logic — tmt/tests/booted/test-extensions-separate-var.nu:424-442 (e8e50d6): Units that aren't bootc's are blamed on the last extension merged, which is a heuristic: an unrelated flaky unit in CI would be reported as [systemd-confext] or [systemd-sysext]. A unit that fails only after this check runs goes unnoticed, which errs towards passing rather than flaking.
2. note · logic — tmt/tests/booted/test-extensions-separate-var.nu:91-95 (e8e50d6): Operator decision: bootc-publish-rhsm-facts.service failing on a read-only /etc is a real bootc bug that is tolerated here so c10 stays green. The intended exit is a small bootc fix (ConditionPathIsReadWrite=/etc, or skipping on EROFS in crates/lib/src/rhsm.rs) and then dropping this list.
3. look-closely · logic — tmt/tests/booted/test-extensions-separate-var.nu:278-345 (e8e50d6): Rewrites the ESP while the system is booted. sfdisk -N keeps the partition GUID (checked), mkfs.vfat -i keeps the volume ID, and the files are copied back. The VMs boot with uefi-insecure firmware, so the reboots really go through the rewritten ESP. An interruption between the umount and the copy-back would leave the VM unbootable, which shows up as a reboot timeout.
4. note · logic — tmt/tests/booted/test-extensions-separate-var.nu:176-191 (e8e50d6): Operator decision: on systemd < 257, the c9 /usr label from each merge path is only printed as KNOWN ISSUE. A RHEL backport would keep the version at 252, so the TODO is the only exit plan.
5. note · test-gap — tmt/tests/booted/test-extensions-separate-var.nu:412-421 (e8e50d6): A merged confext makes /etc read-only from before sysinit.target, so only boot-time writers to /etc are exercised. A bootc upgrade or switch with extensions merged is declared out of scope in the header.
6. note · logic — tmt/tests/booted/test-extensions-separate-var.nu:248-266 (e8e50d6): Copies the plan directory (the common prefix of TMT_TREE and TMT_PLAN_DATA) and TMT_SCRIPTS_DIR onto the new /var. That works with tmt 1.80 but assumes tmt needs nothing else under /var after the reboot.

Safe to skim

• tmt/tests/tests.fmf:304-313: generated by cargo xtask update-generated from the .nu header
• tmt/plans/integration.fmf:462-475: generated by cargo xtask update-generated from the .nu header

[jmarrero-bot]

Review guide for head 31c8fe85f4: where to look closely and what is safe to skim. It is advice from the bot's reviewer and doesn't replace reading the diff; the review app walks it.

One commit that adds tmt plan 70, gated to centos-9 and centos-10 in image mode on the ostree backend. The test shrinks the VM's ESP from 512M to 128M (the same technique as test-composefs-xbootldr.nu) and puts an empty XFS /var partition in the freed space, mounted from /etc/fstab. On it, a directory sysext (and, on systemd 254 and later, a directory confext; if that is missing there, the test fails) is merged and unmerged from a shell. It is then replaced under a live overlay and refreshed, with v1 to v2 asserted, and finally merged by the services at boot. That final boot must settle (is-system-running --wait, up to 300s) with no failed units. Two KNOWN ISSUEs are operator decisions with TODOs: the systemd 252 /usr relabel on c9, and bootc-publish-rhsm-facts.service failing on the read-only /etc a confext gives on c10. The deliberately broken variants (wrong extension-release, no refresh, confext missing, a unit that fails late) all fail with tagged messages. release_esp's automount path is not exercised by either image. The fmf files are generated.

Hotspots

1. look-closely · logic — tmt/tests/booted/test-extensions-separate-var.nu:449-462 (31c8fe8): The refresh check depends on the overlay keeping the moved v1 directory until refresh. The run without refresh confirms v1 stays visible, so v2 after refresh really comes from the refresh. That negative run fails at the sysext, so the confext's refresh is covered only by the passing run.
2. look-closely · logic — tmt/tests/booted/test-extensions-separate-var.nu:480-503 (31c8fe8): Waits up to 300s for the boot to settle, then fails on any failed unit. Units that aren't bootc's, and a boot that never settles, are blamed on the last extension merged, which is a heuristic. A flaky unit unrelated to extensions in CI would be reported as [systemd-confext] or [systemd-sysext].
3. look-closely · logic — tmt/tests/booted/test-extensions-separate-var.nu:306-323 (31c8fe8): Stops only the automounts whose mount unit's What resolves to the ESP, leaving ostree's boot.mount bind alone. Neither the c9 nor the c10 image has an ESP automount, so the stopping branch has never run, and only the plain findmnt/umount loop is exercised.
4. note · logic — tmt/tests/booted/test-extensions-separate-var.nu:325-390 (31c8fe8): Rewrites the ESP while the system is booted, keeping its PARTUUID (checked) and vfat volume ID. The VMs boot with uefi-insecure firmware, so the reboots really go through the rewritten ESP. An interruption between the umount and the copy-back leaves the VM unbootable, which shows up as a reboot timeout.
5. note · logic — tmt/tests/booted/test-extensions-separate-var.nu:97-103 (31c8fe8): Operator decisions: the c9 /usr relabel tolerated per merge path on systemd < 257, and bootc-publish-rhsm-facts.service tolerated when a confext is merged. Both have TODOs. The rhsm one is a real bootc bug, with a fix proposed in the PR thread.
6. note · error-handling — tmt/tests/booted/test-extensions-separate-var.nu:148-160 (31c8fe8): A missing systemd-sysext always fails, and a missing systemd-confext fails on systemd 254 or later. The catch block deliberately avoids calling this function, so cleanup cannot fail the same way.

Safe to skim

• tmt/tests/tests.fmf:304-313: generated by cargo xtask update-generated from the .nu header
• tmt/plans/integration.fmf:462-475: generated by cargo xtask update-generated from the .nu header

[jmarrero-bot]

Fixed four review findings in 31c8fe8 (still one commit, test files only); the reviewer re-ran both distros and says ship.

• Refresh now replaces a merged extension with v2 and asserts that refresh shows v2. Before, it only ran after an unmerge, so it never touched a live overlay. A run that skips the refresh fails with holds '... v1', expected '... v2'.
• Missing systemd-confext on systemd 254+ now fails [systemd-confext] instead of printing SKIP and passing. c9 (252) still skips it.
• Failed units: the final boot now waits up to 300s for systemctl is-system-running --wait before checking. A unit that fails 60s into boot is now caught.
• ESP: before rewriting the ESP, the test stops any automount/mount units that point at it. It leaves ostree's /boot bind mount alone. Neither image has such a unit, so no run exercised this.

c9s and c10s pass; each of the deliberately broken variants fails with a tagged message; just validate passes. Logs: https://gist.github.com/jmarrero-bot/ebbba1edfa9d7a805f304a789afec21b