Skip to content

Stop subprocess tests timing out when the machine is busy - #191

Merged
arzafran merged 1 commit into
mainfrom
fix/test-runprocess-thread-starvation
Jul 29, 2026
Merged

Stop subprocess tests timing out when the machine is busy#191
arzafran merged 1 commit into
mainfrom
fix/test-runprocess-thread-starvation

Conversation

@arzafran

Copy link
Copy Markdown
Member

What this does

unit-tests on main keeps failing on subprocess tests that time out after 90 seconds running scripts that finish in about 15 milliseconds. Two different tests took turns breaking the build today, and re-running did not clear it.

Nothing was actually hanging. The test helper handed the "wait for this subprocess" job to a shared background worker, and that worker stayed occupied for as long as the subprocess lived. The suite runs in parallel, so enough of these at once fills the shared pool, the waiting job never gets a turn, and the test gives up on its own timer long after the subprocess already exited.

The wait is now driven by the notification the system already sends when a process ends, which occupies nothing.

Summary

  • Both private runProcess helpers in programaTests/WorkspaceRemoteConnectionTests.swift (:17, :1473) previously did DispatchQueue.global(qos: .userInitiated).async { process.waitUntilExit(); exitSignal.signal() }, blocking a pooled worker for the child's entire lifetime.
  • Replaced with process.terminationHandler = { _ in exitSignal.signal() }, set before run(). Foundation delivers it on its own private queue and blocks no shared thread.
  • Everything else is unchanged: terminate() on timeout, the 1s grace wait, both pipes read with readDataToEndOfFile(), and process.terminationStatus as the returned status.
  • runProcess in programaTests/TabManagerUnitTests.swift:90 is deliberately untouched — it has no timeout and calls waitUntilExit() on the calling thread, so it cannot produce this failure.

Evidence

  • testRemoteRelayMetadataCleanupScriptPreservesDifferentSocketAddr fails at exactly its 90s timeout. Running the identical script it invokes, with the same non-matching socket_addr, completes in 0.015s with the exact filesystem outcome the test asserts.
  • Failures land precisely on the configured limit, move between tests run to run, and only appear under full-suite parallel load — all consistent with the waiter never being scheduled rather than the child hanging.
  • Four failures observed today, every one of them in a runProcess-based test in this class.

Test plan

  • xcodebuild -scheme programa-unit build-for-testing succeeds
  • unit-tests passes on this PR
  • unit-tests stays green across several subsequent main runs — this is the real check, since a single green run cannot distinguish a fix from a lucky scheduling outcome

Worth flagging separately, pre-existing and not addressed here: both helpers read the pipes only after the process exits, so a child producing more than the pipe buffer (~64KB) would deadlock. No current test produces that much output.

Two test helpers waited for a subprocess by handing the wait off to a
shared background worker. That worker is taken from a pool everyone else
is also using, and it stays occupied for as long as the subprocess runs.
When the whole suite runs at once the pool fills up, the wait never gets
a turn, and the test gives up on its own timer long after the subprocess
has actually finished.

The result was tests failing after a minute and a half on scripts that
finish in fifteen milliseconds. Two of them took turns breaking the build
on main today.

Now the wait is driven by the notification the system already sends when
a process ends, which occupies nothing.
@arzafran
arzafran merged commit 61eb489 into main Jul 29, 2026
9 of 10 checks passed
@arzafran
arzafran deleted the fix/test-runprocess-thread-starvation branch July 29, 2026 16:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant