Skip to content

ci: faster pull request runs: split slow Linux jobs, cancel on compile failure, Windows cache on Dev Drive - #344838

Merged
Logan Ramos (lramos15) merged 4 commits into
mainfrom
lramos15/pr-ci-speed
Oct 29, 2026
Merged

Logan Ramos (lramos15) merged 4 commits into
mainfrom
lramos15/pr-ci-speed

Conversation

@lramos15

@lramos15 Logan Ramos (lramos15) commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

Makes pull request runs faster and frees scarce macOS and Windows runners. Three independent commits; reviewing them one at a time is easiest. The Linux cache warmer fix is separate, in #344837.

Why

Measured over about 450 successful Code OSS runs from Oct 26 to 28:

  • Linux jobs set the run time. The median run takes 16.9 minutes. Linux / Remote and Linux / Electron are most often the last job to finish, because each runs two suites one after the other (about 11 minutes of tests).
  • Failed runs keep using machines. Compile & Hygiene fails in about 5% of runs, at about 2 minutes in for hygiene and ESLint failures. The test jobs of those runs keep going and use about 10 macOS and 15 Windows runner hours a day.
  • Windows cache misses cost 12 minutes per job. For about 22 minutes after every dependency change on main, pull request runs miss the Windows node_modules cache, and each Windows job installs the dependencies itself (about 12 minutes on the Dev Drive). The warming job installs on the OS disk, which takes 16 to 20 minutes, so it saves the cache late.

What changed

  1. ci: split the slowest Linux PR test jobs (pr-linux-test.yml). Linux runners have no queue, so:

    • Electron becomes Electron (node.js) and Electron (extension host). A new INTEGRATION_TEST_SUITE env selects one, as UNIT_TEST_SUITE does for the Electron-Unit jobs.
    • Remote becomes Remote (integration) and Remote (smoke), as on Windows.
    • The Electron smoke tests use three shards instead of two, as on Windows.

    The longest Linux job drops from about 13.5 to about 9 minutes. The workflow tests and the skills that name the jobs and artifacts are updated.

  2. ci: cancel the pull request run when Compile & Hygiene fails (pr.yml).

    • Compile & Hygiene reports whether one of its checks failed, as opposed to its setup, which a rerun fixes: type check, Markdown editor checks, hygiene/ESLint, compile, Codex sync, cyclic dependencies.
    • A new Cancel Run job then runs gh run cancel. It is separate so that pull request code never runs with an actions: write token.
    • Pull requests from forks get a read-only token, so their runs are not cancelled.
    • PR Gate lists the job, which its test requires, but ignores its result: Compile & Hygiene already fails the gate.
  3. ci: install the Windows node_modules cache on a Dev Drive (pr-node-modules.yml). The Windows warming job moves its workspace to a Dev Drive with the same step as pr-win32-test.yml. A new test keeps the two steps the same.

Reviewer notes

How to test

  • The Linux checks of this PR show the new jobs.
  • Cancellation was checked on this PR with a temporary commit containing a type error (since removed). In run 40763077190:
    • Compile & Hygiene failed at 20:29:31 with Type 'string' is not assignable to type 'number'.
    • Cancel Run logged ✓ Request to cancel workflow 40763077190 submitted. 8 seconds later.
    • 35 jobs were cancelled, including every macOS and Windows test job.
    • PR Gate failed, reporting compile did not succeed (result: failure).

Locally:

  • cd build && node --test lib/test/detectMissingCaches.test.ts lib/test/detectChanges.test.ts lib/test/agentHostTestSelection.test.ts: 33 passed, 0 failed. The full node --test 'lib/**/*.test.ts' also passed: 587 passed, 0 failed.
  • npm run eslint on the changed test files passed, and npm run typecheck in build passed.
  • actionlint 1.7.7 reports nothing new compared with main, apart from its known limits with YAML anchors and custom runner labels.

The Windows cache warming job installs the dependencies on the OS disk,
which takes 16 to 20 minutes, against about 12 on the Dev Drive that the
Windows test jobs use. Every pull request run started before the job
saves the cache, about 20 minutes after each dependency change on main,
has each of its Windows jobs install the dependencies itself. Move the
workspace to a Dev Drive with the step of pr-win32-test.yml, and test
that the two steps stay the same.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When hygiene, ESLint, the compilation or another check of Compile &
Hygiene fails, the change has to be fixed and pushed again, but the test
jobs keep running: about 10 macOS and 15 Windows runner hours a day go
to runs whose Compile & Hygiene already failed. Hygiene and ESLint
failures, most of them, are known about 2 minutes into the run.

Compile & Hygiene now reports whether one of its checks failed, as
opposed to its setup, which a rerun of the job fixes. A new Cancel Run
job then cancels the run. It runs apart from Compile & Hygiene so that
the code of the pull request never runs with a token that can cancel
workflow runs. The token of a pull request from a fork is read-only, so
those runs are not cancelled. PR Gate lists the job but ignores its
result: Compile & Hygiene already fails the gate.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The Linux Remote and Electron jobs run two test suites one after the
other, about 11 minutes of tests, and most pull request runs finish when
they do. The Electron smoke test shards take about 9 minutes of tests.
Linux machines are plentiful, so:

- Electron becomes Electron (node.js) and Electron (extension host). A
  new INTEGRATION_TEST_SUITE env selects one of the two, like
  UNIT_TEST_SUITE does for the Electron-Unit jobs.
- Remote becomes Remote (integration) and Remote (smoke), as on Windows.
- The Electron smoke tests use three shards instead of two, as on
  Windows.

The longest Linux job drops from about 13.5 to about 9 minutes. PR Gate
already reports every job of the Linux workflow. The skills that name
the jobs and their artifacts are updated.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 28, 2026 20:22

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The cancellation probe assigns a string to a number, causing TS2322 and preventing Compile & Hygiene from passing.

1 open issue
What changed in this PR

This PR aims to shorten VS Code pull request checks and free runners sooner by increasing Linux test parallelism, cancelling runs after compilation or hygiene check failures, and accelerating Windows cache warming.

Changes:

  • Split Linux Electron and Remote test jobs, and increase Electron smoke testing to three shards.
  • Add an isolated cancellation job for same-repository pull requests, plus a temporary compilation-failure probe.
  • Move Windows cache warming to a Dev Drive and update workflow tests and troubleshooting documentation.
File Description
src/​vs/​base/​common/​numbers.ts Adds a deliberate type error to test cancellation.
build/​lib/​test/​detectMissingCaches.test.ts Checks parity between Windows Dev Drive setup steps.
build/​lib/​test/​detectChanges.test.ts Updates expected job names and cancellation-job coverage.
build/​lib/​test/​agentHostTestSelection.test.ts Adapts assertions to split Linux integration jobs.
.github/​workflows/​pr.yml Reports failed checks and adds isolated run cancellation.
.github/​workflows/​pr-node-modules.yml Moves Windows cache warming onto a Dev Drive.
.github/​workflows/​pr-linux-test.yml Splits slow test jobs and adds a third smoke shard.
.github/​skills/​smoke-tests/​SKILL.md Documents revised sharding and remote smoke jobs.
.github/​skills/​fix-ci-failures/​SKILL.md Updates artifact names and download examples.

Review effort: Balanced

Comment thread src/vs/base/common/numbers.ts Resolved
@lramos15
Logan Ramos (lramos15) merged commit a369c6e into main Oct 29, 2026
57 of 101 checks passed
@lramos15
Logan Ramos (lramos15) deleted the lramos15/pr-ci-speed branch October 29, 2026 00:29
@vs-code-engineering vs-code-engineering Bot added this to the 1.145.0 milestone Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants