[Michael Lively (Yoyokrazy)]

Moves the core PR checks off the 1ES self-hosted pools and onto the GitHub-hosted larger-runner production pools. Tracking: microsoft/vscode-engineering#4273.

Changes

File	Jobs	Before	After
pr.yml	Compile & Hygiene, Copilot - Check Test Cache, Copilot - Check Telemetry, Copilot - Test (Linux)	1ES ubuntu-22.04	vscode-large-runners-ubuntu-8-core
pr.yml	Copilot - Test (Windows)	1ES windows-2022	vscode-large-runners-windows-8-core
pr-win32-test.yml	all Windows test jobs	1ES windows-2022	vscode-large-runners-windows-8-core
pr-node-modules.yml	Compile, Linux, Copilot (Linux)	1ES ubuntu-22.04	vscode-large-runners-ubuntu-8-core
pr-node-modules.yml	Windows, Copilot (Windows)	1ES windows-2022	vscode-large-runners-windows-8-core
pr-linux-test.yml	all Linux test jobs	standard ubuntu-24.04 (4-core)	vscode-large-runners-ubuntu-8-core
pr-linux-cli-test.yml	Linux / CLI, Linux Copilot Shim / CLI	standard ubuntu-24.04 (4-core)	vscode-large-runners-ubuntu-8-core
detect-missing-caches/detect.ts (+ test)	Electron cache version	1ES home dirs (/home/cloudtest, C:\Users\cloudtest)	GitHub-hosted home dirs (/home/runner, C:\Users\runneradmin)

The diff is almost entirely runs-on lines so it rebases cleanly against #342735.

Why detect.ts changes: actions/cache puts the absolute cache path into the cache version. setup-electron caches $HOME/.cache/electron (Linux) and %LOCALAPPDATA%\electron\Cache (Windows), so the Electron cache that pr-node-modules.yml saves from 1ES (/home/cloudtest) can't be restored on GitHub-hosted runners (/home/runner). The Linux test jobs have been missing this cache since they moved to ubuntu-24.04. After this change the warming jobs and the test jobs run on the same images, and detect.ts has to compute the version from the new paths, or the Linux and Windows warming jobs would run on every push to main.

Assumptions

1. Every 1ES runs-on (self-hosted, 1ES.Pool=…, JobId=…) moves to the 8-core production pools. The JobId labels were 1ES-specific and are dropped.
2. The PR check jobs that do real work on standard runners also move to the 8-core pools: pr-linux-test.yml and pr-linux-cli-test.yml.
3. Left unchanged:
   • changes (Detect Changes), required-checks, and the pr-node-modules caches job: they finish in seconds.
   • The chat-lib jobs: their required check names contain ubuntu-latest / windows-latest.
   • macOS (macos-26-xlarge): fixed size.
   • The 16/32/64-core pools: used for experiments elsewhere.
4. node_modules cache keys stay the same. Caches built on the old images (Ubuntu 22.04 with the pinned sysroot, Windows Server 2022 with VS 2022) restore onto the newer images (Ubuntu 24.04, Windows Server 2025). That direction is compatible, and the Linux test jobs already do it today (1ES 22.04-built node_modules-linux on ubuntu-24.04). Changing the key would force every PR to run npm ci on every job until main re-warms the caches (about 12.5 min per Windows job). The keys include every package-lock.json, so the old-image entries are replaced with the next dependency change after this merges. The cache-miss (npm ci) path was validated on both new pools with a temporary key-prefix commit, which was reverted (see Validation).
5. pr-node-modules.yml only runs on push to main, so its move takes effect after merge. Until main re-warms the caches, Windows PR jobs miss the Electron cache (different cache path) and download Electron instead. After the merge, both OSes get Electron cache hits; the Linux test jobs have been missing it since they moved to ubuntu-24.04.

Follow-ups (not in this PR)

Other workflows still on standard runners: codeql.yml, component-fixtures.yml, telemetry.yml, monaco-editor.yml, copilot-setup-steps.yml (vscode-large-runners), chat-perf.yml, sessions-e2e.yml, css-order-scan.yml, chat-lib-package.yml, and the chat-lib jobs in pr.yml (the required check names need to change first).

Validation

Run	node_modules cache	Result
39522227766 (782bbf2)	hit on all jobs	❌ Windows / Electron (Agent Host E2E step hit its 30-min timeout), ❌ Windows / Electron-Unit (known Mission Control WPS EPERM flake, same as #341529)
39539559733 (4eaff64, temporary key-prefix change, reverted in 6232879)	miss on all jobs (npm ci on both pools)	✅ all green
39544817211 (6232879, final; same tree as 782bbf2)	hit on all jobs	attempt 1: ❌ Windows / Electron (one Agent Host E2E request timed out); attempt 2 (re-run): ✅ all green (Agent Host E2E 26.6 min)

• node --test build/lib/test/detectMissingCaches.test.ts: passed locally (4/4). Compile & Hygiene also runs the build typecheck and the build-script tests.
• Linux 8-core: every job passes, and is the same speed or faster than the current runners.
• Windows 8-core: everything passes except Agent Host E2E when all 6 suites run on one runner. It took 26.5–32 min and failed in 3 of 6 runs; 1ES had a 15.8-min median and 5.9% failures over 269 other-PR runs. With 2 suites per runner it takes 7.9–9.1 min and passed 6 of 6 jobs. So the fix is to split it across 3 runners on top of ci: run Agent Host E2E tests in their own job on Windows #342735; details are in the results comments. Windows / Electron shouldn't move until that lands. This blocks marking the PR ready.
• The temporary diagnostic workflows and the key-prefix change are reverted. The final tree equals 782bbf2.

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

1 open issue

• Hardcoding absolute home-directory paths (/home/runner, C:\\Users\\runneradmin) makes… · New

What changed in this PR

Moves core PR checks from 1ES self-hosted pools to GitHub-hosted 8-core “vscode-large-runners” pools, and updates cache-detection logic/tests to align Electron cache versions with GitHub-hosted runner home directories.

Changes:

• Updated multiple PR workflows to run on vscode-large-runners-*-8-core runners.
• Updated Electron cache path assumptions in detect-missing-caches to match GitHub-hosted runner home directories.
• Updated detectMissingCaches test runner labels to match the new runs-on values.

File	Description
build/​lib/​test/​detectMissingCaches.test.ts	Updates runner label expectations in tests to match new workflow runs-on labels.
.github/​workflows/​pr.yml	Moves compile/hygiene and Copilot PR jobs to GitHub-hosted 8-core runner pools.
.github/​workflows/​pr-win32-test.yml	Moves all Windows test jobs to GitHub-hosted 8-core Windows runner pool.
.github/​workflows/​pr-node-modules.yml	Moves warm/cache-related compile/Linux/Windows/Copilot jobs to GitHub-hosted 8-core pools.
.github/​workflows/​pr-linux-test.yml	Moves Linux test jobs from standard runners to GitHub-hosted 8-core pool.
.github/​workflows/​pr-linux-cli-test.yml	Moves Linux CLI test jobs from standard runners to GitHub-hosted 8-core pool.
.github/​actions/​detect-missing-caches/​detect.ts	Updates Electron cache absolute paths to GitHub-hosted home directories for stable cache-version computation.

_{Review effort: Lite}

[Michael Lively (Yoyokrazy)]

Results: PR checks on the GitHub-hosted 8-core pools

Summary: Linux works well on vscode-large-runners-ubuntu-8-core. Every Linux job is as fast or faster: Remote is 4.3 min faster, Electron and Electron-Unit are about 2 min faster. Windows works on vscode-large-runners-windows-8-core, including WSL2 and native modules built with VS 2026, and most Windows jobs are within ±1.5 min of 1ES. Windows Agent Host E2E is the blocker. On these runners it takes about 2x as long (27.9 vs 14.4 min) and failed 2 of 3 runs on 20–30 s request timeouts. The cause is the disk: these runners only have a remote Premium SSD (C:), and 1ES had a local temp SSD (D:). Details and options are below.

Runs

Run	Commit	node_modules cache	Result
39522227766	782bbf2 (this PR)	hit (18/18 jobs)	❌ Windows / Electron (Agent Host E2E step hit its 30-min timeout), ❌ Windows / Electron-Unit (known Mission Control WPS EPERM flake, same as #341529)
39539559733	4eaff64 (temporary key-prefix change, now reverted)	miss (18/18 jobs, npm ci)	✅ all green (Agent Host E2E passed in 26.6 min)
39544817211	6232879 (final state; same tree as 782bbf2)	hit (18/18 jobs)	❌ Windows / Electron (Agent Host E2E: 1 test hit a subscribe timeout after 27.9 min)

All three runs used the full Agent Host E2E profile, because workflow changes count as an infrastructure change.

Per-job durations (minutes)

The new-pool median is over runs 1 and 3 (cache hit). The 1ES-era median is over 11 recent successful pr.yml runs on other PRs from Oct 21 (1ES for Compile, Copilot and Windows; standard 4-core ubuntu-24.04 for Linux tests). "Given" is the baseline from the issue (Oct 20, before #342729).

Job	Pool	Given	1ES-era median	Run 1	Run 3	8-core median	Δ	Cache miss: job (npm ci)
Compile & Hygiene	ubuntu-8	8.0	8.0	7.2	9.2	8.2	+0.2	13.6 (5.2)
Copilot - Check Test Cache	ubuntu-8	–	0.6	0.7	0.7	0.7	+0.1	2.2 (1.9)
Copilot - Test (Linux)	ubuntu-8	–	9.0	6.8	7.0	6.9	−2.1	10.8 (1.8)
Copilot - Test (Windows)	windows-8	–	11.4	13.4	11.4	12.4	+1.1	18.8 (7.3)
Linux / Electron	ubuntu-8	14.3	15.3	11.7	14.6	13.2	−2.2	19.7 (4.2)
Linux / Electron-Unit	ubuntu-8	14.9	14.7	13.9	10.7	12.3	−2.4	18.4 (4.2)
Linux / Remote	ubuntu-8	15.8	15.0	9.9	11.6	10.7	−4.3	13.1 (2.9)
Linux smoke shard 1 / 2	ubuntu-8	11.0 / 12.3	11.0 / 11.4	10.3 / 10.3	10.1 / 9.5	10.2 / 9.9	−0.8 / −1.5	12.1 / 13.9
Linux Browser (unit / integration & smoke)	ubuntu-8	–	10.3 / 7.9	9.5 / 5.7	9.9 / 6.3	9.7 / 6.0	−0.6 / −1.9	13.0 / 10.8
Linux / CLI, Linux Copilot Shim / CLI	ubuntu-8	–	2.9 / 0.5 (n=1)	1.6 / 0.6	1.9 / 0.5	1.7 / 0.5	−1.2 / 0	1.9 / 0.5
Windows / Electron (with Agent Host E2E)	windows-8	37.5	32.0 (7 runs where Agent Host E2E ran)	48.7 ❌	42.9 ❌	45.8	+13.8	56.7 (18.7)
↳ Agent Host E2E step			14.4 (9.6–19.0)	30.0 (timeout)	27.9 ❌	27.9 (3 runs, incl. run 2: 26.5 ✅)	+13.5	26.5 ✅
↳ Integration tests step			11.8	12.6	10.5	11.6	−0.2	10.1
Windows / Electron-Unit	windows-8	20.7	19.8	20.5 ❌	16.0	18.3	−1.6	41.7 (23.2)
Windows / Remote	windows-8	15.3	14.3	15.6	12.4	14.0	−0.4	29.5 (18.5)
Windows smoke shard 1 / 2	windows-8	13.5 / 11.0	13.1 / 10.5	15.9 / 12.5	13.4 / 9.5	14.7 / 11.0	+1.5 / +0.5	32.5 (19.7) / 34.4 (24.4)
Windows Browser (unit / integration & smoke)	windows-8	–	10.2 / 9.3	13.4 / 10.6	9.4 / 8.6	11.4 / 9.6	+1.3 / +0.4	33.7 / 27.4

The Linux Agent Host E2E step dropped from 7.3 min (4-core) to 4.8 min.

npm ci on a cache miss (run 2)

Pool	npm ci	1ES / standard baseline (from the issue)
ubuntu-8	2.9–5.2 min (Compile 5.2)	3–5 min
windows-8	18.5–24.4 min (median about 19.7)	about 12.5 min

Both pools build all native modules from scratch. On Windows that's the windows-2025-vs2026 image with VS 2026, which build/npm/preinstall.ts already detects. checkNativeOptionalDeps.ts isn't part of the PR jobs, but every test that loads native modules passed.

Runner queue time (job created → started)

Pool	Median	Max	Jobs
ubuntu-8	0.08 min	0.10 min	39
windows-8	2.0 min	3.4 min	24
1ES ubuntu (baseline)	0.22 min	0.32 min	17
1ES windows (baseline)	0.18 min	0.32 min	80
standard ubuntu-24.04 (baseline)	0.07 min	0.33 min	255

Windows 8-core runners take about 2 min to start when no runner is idle. A runner that was just released starts in about 0.05 min.

Windows Agent Host E2E: root cause

I investigated with temporary workflows on this PR (all removed). They used a handful of short jobs on the 8-core pool and 1ES. No pool configuration was changed.

• Disk layout: the 8-core Windows runner has a single "Virtual_Disk NVMe Premium SSD" C: (440 GB) and no temp disk, so the workspace is C:\a\vscode\vscode. 1ES (D8ds_v5) has the workspace on a local temp SSD (D:\a\_work). Standard windows-latest also has a D: temp disk.
• Ruled out: Defender real-time scanning (off on all three), VBS/HVCI (off), CPU (AMD EPYC 9V74 vs Xeon 8573C, both 4C/8T, with similar single-thread loops), and process spawn cost (30–37 ms for warm node.exe on all three). CPU-bound steps are faster on the new runner: Electron unit tests 9.1 vs 10.8 min, node unit 5.2 vs 6.2.
• Bottleneck: Agent Host E2E runs 6 suites in parallel and does about 700–1,100 small writes/s (about 9 KB each, about 6–9 MB/s). On 1ES that averages 0.5–4 ms per write at 31–52% CPU. On the 8-core runner it averages 7–20 ms per write with a queue depth of 6–7, while the CPU sits at 14–20%. The tests then hit their 20–30 s createSession/createChat/subscribe timeouts. A freshly provisioned VM adds background I/O on the same disk (Defender engine about 1.9 GB read+write, Windows servicing, Search indexer).
• Mitigations tried in-job (Agent Host E2E duration; 1ES ran the same experiment in 10.2 and 16.9 min):

Variant	Result
Control	32 min ❌ / 30 min timeout ❌
--jobs 3	24.9 min ❌
TEMP on a ReFS Dev Drive VHDX	25.8 min ✅ / 35.3 min ❌
Dev Drive mounted at the workspace + TEMP	25.8 min ✅ (checkout 12 s, node_modules restore 59 s vs 3–4 min)
Disk flushes suppressed (IOCTL_DISK_SET_CACHE_SETTING, power-protected)	26.2 min ❌ (fsync 1.4 → 0.5 ms, but the step didn't get faster)
RAM disk via the iSCSI Target ramdisk: feature	not viable: the feature install alone takes 2.6 min, and the loopback portal fails

None of these brings Windows Agent Host E2E back to 1ES speed or makes it reliable, so this PR doesn't include them.

Other platform checks

• Ubuntu 22.04 → 24.04 (Compile, Copilot): no changes needed. The Copilot extension-host tests run fine under xvfb-run without the AppArmor userns workaround.
• Windows Server 2022 → 2025 (windows-2025-vs2026): npm ci builds the native modules with VS 2026. The Copilot Windows job (setup-python 3.12, .NET 10, setuptools) passes on both cache paths.
• WSL2 / nested virtualization: works. wslDevContainer.ps1 imports the WSL2 distro and Windows smoke shard 1 passes (WSL setup 2.8 vs 1.5 min on 1ES).
• Disk layout: workspace on C: (see above). On a cache hit, extracting node_modules takes 2.9–3.8 min vs 1.3 min on 1ES. That's the main reason the other Windows jobs are 0.5–1.5 min slower.
• Electron cache: actions/cache includes the absolute path in the cache version, so the Electron cache that pr-node-modules.yml saved from 1ES (/home/cloudtest, C:\Users\cloudtest) could never be restored on GitHub-hosted runners. The Linux test jobs have been downloading Electron every time since they moved to ubuntu-24.04. This PR moves the warming jobs and updates detect-missing-caches, so after merge both OSes get Electron cache hits.
• node_modules cache key: unchanged (reasons are in the PR description). Old-image caches restored fine on the new images in runs 1 and 3, and the miss path works on both pools (run 2).

Options for Windows Agent Host E2E (need a decision)

1. Use a Windows pool with local temp storage for at least the job that runs Agent Host E2E. That's a pool or VM-SKU decision. I didn't change any pool configuration.
2. Try the 16-core Windows pool for that job: a 600 GB disk means a higher Premium SSD IOPS tier. It's out of scope here because those pools are used for experiments.
3. Make the suites tolerate slower disks on Windows: longer per-request timeouts, or use a RAM-backed temp directory once Windows has a practical way to create one (Linux already uses tmpfs, split storage). Accept a longer Windows long pole. Combined with ci: run Agent Host E2E tests in their own job on Windows #342735, Agent Host E2E would become its own about 32-min Windows job.
4. Separately, consider the Dev Drive workspace for all Windows jobs: it cut node_modules restore from 3–4 min to 59 s. It's not in this PR because it changes the filesystem the Windows tests run on (ReFS), and only Agent Host E2E has been validated on it.

Follow-ups (not in this PR)

codeql.yml, component-fixtures.yml, telemetry.yml, monaco-editor.yml, copilot-setup-steps.yml (vscode-large-runners), chat-perf.yml, sessions-e2e.yml, css-order-scan.yml, chat-lib-package.yml, and the chat-lib jobs in pr.yml (their required check names would need to change first).

[Michael Lively (Yoyokrazy)]

Update: re-run and 1ES comparison

I re-ran the failed job and Windows / Electron passed on attempt 2: Agent Host E2E took 26.6 min, the whole job 38.8 min. All checks are green now.

The Windows Agent Host E2E slowdown is caused by this PR's runner change; it isn't a flake. The comparison below uses every other PR's 1ES Windows / Electron job since Oct 20 that ran the same full Agent Host E2E profile as this PR:

Agent Host E2E step	1ES (n=269)	8-core (n=4)
Duration	median 15.8, p90 18.9, max 24.6 min	26.5, 26.6, 27.9, 30.0 (timeout)
Step failed	5.9%	2 of 4

• All four 8-core runs were slower than the slowest of the 269 1ES runs.
• Two failures out of four at the 1ES failure rate has about a 2% chance.
• In these runs the long pole of the PR checks moves from about 32 to 39–49 min when Agent Host E2E runs.

The Mission Control WPS failure in run 1 (Windows / Electron-Unit) looks like a pre-existing Windows file-locking flake:

• That test has flaked before (Build failed #341529).
• 13 of 525 1ES node unit runs in the same period failed, mostly with the same kind of EPERM/EBUSY error on rename or unlink (CopilotComputerUsePlugin 6×, Session state compatibility 2×).
• It passed in the next two runs.

[Michael Lively (Yoyokrazy)]

Update: Windows Agent Host E2E can run on the 8-core pool if the suites are split across runners

Correction: I wrote earlier that 1ES is faster because it has a local temp SSD. That's wrong: on 1ES, Agent Host E2E's disk traffic goes to C:, not the local SSD. With all 6 suites on one runner, 1ES's C: averages 2.4 ms per write and the GitHub runner's C: 14–18 ms.

Fix: split the suites across runners. With 2 suites per runner on vscode-large-runners-windows-8-core, Agent Host E2E takes 7.9–9.1 min (6 of 6 jobs passed), faster than 1ES today (median 15.8 min). All 6 suites on one runner takes 26.5–32 min, and 3 of 6 runs failed. Three suites per runner already fails the same way. A different OS version (Server 2022), stopping background services, and bigger runners don't help.

Full tables: https://github.com/microsoft/vscode-engineering/issues/4273#issuecomment-6325648086

For this PR: Windows / Electron shouldn't move to the 8-core pool until #342735 (Agent Host E2E in its own job) also splits it into 3 jobs of 2 suites on Windows. Until then, keep the Windows 1ES pool, or merge only the Linux changes.

Addendum: one runner, 2 suites at a time (run 39672938507, --jobs 2, which the runner already supports). This option adds no runners. All 3 runs passed, but they took 17.8, 27.8 and 27.6 min. In the two slow runs, the first two suites ran at normal speed, but the two that started 7–9 min in (Codex, Copilot) ran about twice as slow as in the fast run. So on some runners the disk slows down during a long run.

Option	Extra Windows runners per PR	Agent Host E2E	Result
1ES today	–	median 15.8 min	5.9% of runs failed
GitHub 8-core, 1 runner, all 6 at once (this PR as-is)	0	26.5–32.0 min	3 of 6 failed
GitHub 8-core, 1 runner, --jobs 2	0	17.8–27.8 min	✅ 3 of 3
GitHub 8-core, 3 runners × 2 suites	+3 (+2 on top of #342735's separate job)	7.9–9.1 min	✅ 6 of 6; about 36 runner-minutes vs about 16 today