Skip to content

Make the compatibility check actually check compatibility - #199

Merged
arzafran merged 1 commit into
mainfrom
ci/compat-gate-on-smoke
Jul 29, 2026
Merged

Make the compatibility check actually check compatibility#199
arzafran merged 1 commit into
mainfrom
ci/compat-gate-on-smoke

Conversation

@arzafran

Copy link
Copy Markdown
Member

What this does

This job exists to answer one question: does Programa build and launch on a macOS version other than the one release.yml ships from? It has never answered it.

The full ~1,500-test unit suite ran first and had to pass before the build and smoke steps were allowed to run. Those tests fail on the macos-15 runner for reasons unrelated to the OS, so the steps that matter were skipped on every single run:

10. Run unit tests            :: failure
13. Create virtual display    :: skipped
14. Build app for smoke test  :: skipped
15. Smoke test                :: skipped

That has been the state since at least 21 July. Five successive attempts to quiet the failing tests (#193 timeout scaling, #194 17 methods, #197 7 classes, #198 8 classes) each fixed what was in front of them and revealed the next thing — while the actual compatibility signal stayed unreachable behind them.

The change

Build and smoke now run first and gate the job. Unit tests run afterwards, continue-on-error, and report without deciding.

Two reasons that ordering is right, not just pragmatic:

  1. The unit suite is OS-agnostic logic that already runs on every PR in ci.yml. Re-running it on two more macOS versions produced no unique signal.
  2. It was actively hiding the one result only this job can produce.

Also fixes a coverage gap I found while doing this

matrix.smoke is true only on macos-15. macos-26's entire contribution was running unit tests — it never built or launched the app as a gating step. The new "Build app" step runs on both legs, so each now proves something only it can: macos-15 builds ghostty from Zig and launches the app under a virtual display; macos-26 builds against the macOS 26 SDK with the prebuilt xcframework.

I did not enable smoke on macos-26. It is disabled there for a reason I could not establish (that leg also sets skip_zig: true because "zig 0.15.2 MachO linker can't resolve libSystem on macOS 26"), and turning on a virtual-display smoke test I cannot verify would be guessing.

Details worth reviewing

  • if: failure()if: always() on the crash-report and xcresult-upload steps. With the unit-test step continue-on-error, the job is green even when tests fail, so failure() would never fire and the artifacts would silently stop being collected.
  • A final step surfaces the informational outcome as a ::warning:: annotation, so a failing unit suite is still visible in the run summary rather than buried in a green job.
  • The ordering rationale is a comment in the workflow, not just this PR description. The next person to "tidy up" the step order needs to know why the tests come last.

What this does not do

It does not fix the underlying starvation, the WorkspaceStressProfileTests host crash, or macos-26's intermittent WindowBrowserSlotViewTests failure. Those remain open and are now visible as warnings instead of a permanently red check nobody reads.

Test plan

  • tests/test_ci_unit_test_runner_behavior.sh passes (the existing guard on this runner script)
  • Step order verified: Build app → Create virtual display → Smoke test → Run unit tests (informational) → collect/upload/report
  • Indentation consistent with surrounding steps (6 for - name:, 8 for keys) — no YAML validator available locally, so checked structurally
  • workflow-guard-tests passes
  • macos-15 reaches the smoke test for the first time — this is the real check. Green means compatibility is finally being verified; a smoke-test failure would be genuine, and worth knowing.

This job exists to answer one question: does Programa build and launch on
a macOS version other than the one we ship from. It has never answered it.

The full test suite ran first and had to pass before the build and launch
steps were allowed to run. Those tests fail on the older machine for
reasons that have nothing to do with the operating system, so the build
and launch steps were skipped every single time, for over a week. Five
separate attempts to quiet the failing tests each fixed what was in front
of them and uncovered the next thing.

Building and launching now runs first and decides whether the job passes.
The test suite still runs afterwards and still reports, but no longer
decides, because it is the same suite that already runs on every pull
request and re-running it on another macOS version was never the point.
@arzafran
arzafran merged commit 906451a into main Jul 29, 2026
9 checks passed
@arzafran
arzafran deleted the ci/compat-gate-on-smoke branch July 29, 2026 23:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant