ci: start the PR test jobs from workspace images of main by alexdima · Pull Request #345429 · microsoft/vscode

Alexandru Dima (alexdima) · 2026-11-01T14:43:54Z

Draft. Started by Alexandru Dima (@alexdima), landed by Ladislau Szomoru (@lszomoru). It builds on the move to pnpm 12 (#344791) and on the base key of the node_modules caches (#345840). The temporary validation workflow (workspace-images-validation.yml) is removed before merging. main has no workspace images until this merges, so the test jobs of this pull request warn and set up as before. Merging it builds the first images, since the producer runs when its own files change.

What this changes

The test jobs of pull requests (pr-linux-test.yml, pr-darwin-test.yml, pr-win32-test.yml) start from a workspace image of main instead of an empty folder: a disk image of a workspace whose pnpm dependencies, Electron and Playwright browsers are installed, and whose checkout is at the commit the image was built from.

Building the images: pr-workspace-images.yml builds one per platform when an input of the images changes on main: an input of the install, of the native builds, or of the prepared Electron. It doesn't build them for other changes, since the checkout of a job fetches only what changed since the commit of its image. Like the node_modules caches, it verifies the native optional dependencies. It runs on the runners of the jobs and installs at the exact paths of their workspaces, because pnpm links packages with junctions whose targets are absolute paths on Windows, and records its store in node_modules. The images are kept for 2 days: when no input changes for longer, the jobs set up as before until the next change.

Platform	Image	Download
Linux	ext4 volume without a journal in a sparse file, loop-mounted	1.6 GB
macOS	APFS volume in a sparse ASIF image	1.4 GB
Windows	Dev Drive (ReFS) in a VHDX with 1 MB blocks, attached with cached I/O through the virtual disk API	1.6 GB

All three are restored from 32 MB zstd chunks that download, decompress and write in parallel, without their pages of zeros where the file can be sparse. On Linux, the loop device attaches the file while the image downloads, because attaching a file waits for the kernel to write it to the disk. The kernel then writes the image while the job goes on.

Using an image: a job downloads image.ts for its commit from the API, mounts the newest image of main at the parent folder of its workspace, and checks out with clean: false. The shallow fetch sends a tag at the image's commit and downloads only what changed. Git compares only sizes and modification times, which the images keep, so the checkout writes only the files that changed.
Installing: the job compares the install-state hashes that build/npm/postinstall.ts saved in the image with its checkout (build/npm/installStateHash.ts). It runs its existing install steps only when an input of the install changed, and pnpm then installs only what changed. The pnpm store path is pinned, since pnpm reinstalls everything when it differs.
Fallback: without an image, or on any error, the job warns and sets up as before. It also skips an image whose commit has other inputs of the base key of the node_modules caches (computeNodeModulesBaseCacheKey in build/azure-pipelines/common/cacheKeys.ts) than its own: the salt of the CI caches, the pnpmfiles, the Node.js versions, the pnpm settings of the install roots, and the nativeBuild and packageManager fields of package.json. Installing on top of such an image wouldn't build its native modules again, for example after an Electron update. Before downloading, image.ts compares these inputs between the two commits in one GraphQL request, and workspaceImage.test.ts derives them from the base key, so that the two can't drift apart.
Electron: setup-electron keeps .build/electron when it was prepared from the same inputs, using a stamp of the inputs of pnpm run electron, including the root package.json that it packages and the pnpm-lock.yaml that pins @vscode/gulp-electron.
Playwright: the jobs that use an image skip the cache of the Playwright browsers (ci: cache Playwright browsers for pull request browser tests #343387), as the image holds the browsers.
Trust: a job only uses artifacts of push or manual-dispatch runs of this repository on main, since any pull request could upload an artifact of the same name. Images carry no credentials. The three test jobs of pr.yml get actions: read to find and download the artifacts.

See .github/workflows/workspace_image/README.md for the details.

Results

Without dependency changes, about 9 in 10 test jobs, a job sets up in about half the time or less:

	`main` today, cache hit	With the image	Saved per job
Linux	41 s	21 s	20 s
macOS	61 s	17.5 s	43.5 s
Windows	114 s	57.5 s	56.5 s

These are the sums of the medians of the setup steps that differ between the two setups.

main today: 532 test jobs (248 Linux, 146 macOS, 138 Windows) of the pull request runs of pr.yml created between November 3, 21:05 UTC and November 4, 09:00 UTC, after ci: extract the node_modules cache in parallel parts #345838, ci: start pull requests without a node_modules cache from the cache of other dependencies #345840, ci: split the slowest Windows PR test jobs #345835 and ci: cache Playwright browsers for pull request browser tests #343387 landed, that hit a node_modules cache. 58 more jobs of these runs installed dependencies.
With the image: 18 Linux and 12 macOS jobs of the 6 validation runs of November 2, and 8 Windows jobs of the last 4, which have the current Windows restore.
Both count the job types of those validation runs, the Electron and browser unit test jobs. A job sets up either Electron or Playwright, so each one takes a few seconds less than these sums.

Medians per step, in seconds (main today / with the image)

Step	Linux	macOS	Windows
Download the workspace image scripts	- / 0	- / 1	- / 1.5
Setup Node.js	1 / 1	1 / 0.5	9 / 5
Move the workspace to a Dev Drive	-	-	18 / -
Restore the workspace image	- / 13	- / 10	- / 34
Checkout	13 / 2	14 / 2	21 / 7
Compare the dependencies with the workspace image	- / 0	- / 0	- / 0.5
Setup pnpm	3 / 4	3 / 3	6 / 7.5
Restore node_modules cache	16 / -	34 / -	45 / -
Setup Electron	5 / 0	5 / 0.5	12 / 1
Playwright (cache and install)	3 / 1	4 / 0.5	3 / 1

With a dependency change, the image numbers come from a change of one package in two install roots (@vscode/codicons in the workspace and remote/web), and the numbers of main from whatever real pull requests changed, after restoring the partial cache of #345840, so this comparison is rough:

	`main` today, then install	With the image, then install
Linux	76.5 s	33 s
macOS	87 s	32 s
Windows	152 s	81 s

main today: the 24 Linux, 12 macOS and 22 Windows jobs of the runs above that installed. With the image: 6, 6 and 4 dependency-change jobs of the same validation runs. On top of the image, the install took 9-13 s on Linux, 11-14 s on macOS and 24-45 s on Windows, and installed the right versions.

Where the restore spends its time:

Linux and macOS: downloading and decompressing the image. Writing it doesn't add to that: on Linux, the kernel writes the image to the disk during the next steps, which took as long as before.
Windows: writing the 8.5 GB VHDX, at 230-400 MB/s depending on the runner, while the chunks arrive faster. A VHDX can't be sparse, and a Dev Drive needs at least 50 GB, whose ReFS metadata alone takes 1.9 GB. image.ts creates the VHDX as a temporary file (_O_SHORT_LIVED) and starts the download at once. Meanwhile vhd.ps1 starts, as the job's first PowerShell, which takes 1-15 s, and compiles the virtual disk API.

With every test job at once: the validation runs of the last two commits, 41865808587 and 41875488110, each ran all 34 test jobs of the three workflows, and 33 of them used the image: the WSL smoke test shard keeps today's setup. The restores took 11-34 s on Linux, 15-41 s on Windows and 10-13 s on macOS, apart from two macOS smoke test shards of the first run whose writes into the image were about 3 times slower: 41 s, and 108 s for the one where attaching the image also took a minute.

The images take 3 to 7.5 minutes to build per platform.

Follow-ups

Linux apt packages: Setup system services now takes the longest of the setup steps on Linux (11-54 s, depending on the mirror). Its .deb files could ship in the image.
Linux image size: the sysroots and compilers of setup-env.sh, about 1 GB of the image, are only needed when a dependency changes, and could be downloaded then instead.
pnpm and Node.js: Setup pnpm (3-8 s) and Setup Node.js on Windows (5-9 s at the median) could come from the image. With the image, Setup pnpm takes about as long as the checkout.
Other jobs: the other jobs of pr.yml that restore node_modules (Compile, Monaco Editor, Copilot) could use the images too.
WSL shard: the WSL smoke test shard on the 1ES pool keeps today's setup.

How to test

workspace-images-validation.yml (temporary) builds the images of this branch, runs every test job of the three test workflows with them through the workspace_image_branch input, and brings each image to the dependency change above.

The validation runs of the last two commits, 41865808587 and 41875488110, on top of main: all 40 jobs passed in each, including the integration, smoke, remote and Agent Host E2E jobs, which hadn't run on an image before.
cd build && node --test lib/test/workspaceImage.test.ts lib/test/detectMissingCaches.test.ts lib/test/detectChanges.test.ts lib/test/agentHostTestSelection.test.ts: passed, 34 tests. workspaceImage.test.ts fails when image.ts misses an input of the base key or compares one that it doesn't hash: checked by adding .npmrc and dropping packageManager.
The comparison of image.ts, run locally against the API: the same commit, and a package.json change outside its native fields, kept the image; build: move the native build targets from .npmrc to package.json #345839, which moved the native build targets to package.json, didn't, and named the pnpmfiles, build/npm/pnpm/nativeBuildEnv.ts and the nativeBuild fields.
./node_modules/.bin/tsc --noEmit --strict --module nodenext --moduleResolution nodenext --target es2024 --lib es2024,dom --types node --skipLibCheck --erasableSyntaxOnly --noUnusedLocals --noUnusedParameters .github/workflows/workspace_image/image.ts: passed.
./node_modules/.bin/eslint .github/workflows/workspace_image/image.ts build/lib/test/workspaceImage.test.ts: passed.
actionlint -shellcheck= .github/workflows/pr-workspace-images.yml .github/workflows/pr-linux-test.yml .github/workflows/pr-darwin-test.yml .github/workflows/pr-win32-test.yml .github/workflows/pr.yml .github/workflows/workspace-images-validation.yml (actionlint 1.7.12): passed, apart from the custom runner labels.
Earlier, by Alexandru Dima (@alexdima): pack, unpack (sparse and not) and seed of image.ts locally, and a checkout of a seeded copy, which wrote only the changed files; the contents API serving image.ts at the merge commits of open pull requests from forks and from this repository; and this pull request's own checks, which found no image of main and set up as before.

🤖 Authored with GitHub Copilot on behalf of Alexandru Dima (@alexdima), and updated on behalf of Ladislau Szomoru (@lszomoru).

Copilot

🟡 Changes recommended

Incomplete cache invalidation and missing native optional-dependency validation can cause jobs to reuse stale Electron preparations, dependency trees, or libc++ assets, or consume incomplete workspace images.

4 open issues

What changed in this PR

This PR builds on the pnpm migration to reduce VS Code PR test setup time by restoring preinstalled workspace images of main, with fallback to the existing installation flow.

Changes:

Adds cross-platform image production, compression, restoration, and temporary validation workflows.
Updates test jobs to reuse images and compare dependency state before installing.
Adds prepared-Electron stamps, artifact-read permissions, documentation, and change-detection updates.

File	Description
`build/ripple/vscode.jsonc`	Includes image tooling in test change detection.
`build/lib/test/detectMissingCaches.test.ts`	Updates Windows Dev Drive step-order assertions.
`.github/workflows/workspace-images-validation.yml`	Adds temporary cross-platform image validation.
`.github/workflows/workspace_image/vhd.ps1`	Creates, mounts, and detaches Windows images.
`.github/workflows/workspace_image/Vhd.cs`	Provides Windows virtual disk API bindings.
`.github/workflows/workspace_image/README.md`	Documents image usage, formats, and trust constraints.
`.github/workflows/workspace_image/image.ts`	Implements image packing, restoration, and dependency checks.
`.github/workflows/pr.yml`	Grants test jobs artifact-read permissions.
`.github/workflows/pr-workspace-images.yml`	Builds and publishes images for three platforms.
`.github/workflows/pr-win32-test.yml`	Restores Windows images while preserving the WSL exception.
`.github/workflows/pr-linux-test.yml`	Integrates Linux image restoration and conditional installation.
`.github/workflows/pr-darwin-test.yml`	Integrates macOS image restoration and conditional installation.
`.github/workflows/node_modules_cache/README.md`	Documents prepared-Electron reuse.
`.github/actions/setup-electron/action.yml`	Uses input stamps to skip redundant Electron preparation.

_{Review effort: Balanced}

github-actions · 2026-11-02T12:18:18Z

Base: becf53f4 Current: 6e19f1e5

~~No screenshot changes.~~

@alexdima

The test jobs of pull requests mount a disk image of a workspace of main instead of an empty folder: its pnpm dependencies, Electron and Playwright browsers are installed, and its .git holds the commit it was built at. The checkout reuses the .git and fetches only what changed, and the jobs only run their install steps when an input of the install (build/npm/ installStateHash.ts) changed, which pnpm then brings up to date on the node_modules of the image. pr-workspace-images.yml builds the images on main, at the paths of the workspaces of the jobs, as pnpm links packages with absolute junctions on Windows and records its store in node_modules: an ext4 volume in a sparse file on Linux, an APFS volume in an ASIF image on macOS, and a Dev Drive VHDX on Windows, which restore in parallel chunks. The jobs only take images of pushes, schedules and dispatches of this repository whose native build inputs match their commit, and fall back to the node_modules cache and installing otherwise. setup-electron keeps an Electron that was prepared from the same inputs. The 23 commits of #345429, squashed and rebased onto main. 🤖 Authored with GitHub Copilot on behalf of @alexdima. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…s caches #345839 moved the native build targets from the .npmrc files to the nativeBuild field of package.json, so a job now compares the inputs of computeNodeModulesBaseCacheKey (#345840) between its commit and the commit of a workspace image: the salt, the pnpmfiles, the Node.js versions, the pnpm settings of every install root, and the nativeBuild and packageManager fields of package.json, which the restore reads in the same GraphQL request. workspaceImage.test.ts derives these inputs from the base key, so that they can't drift apart. - The producer also builds the images when package.json, remote/package.json or extensions/copilot/.nvmrc change. - The test jobs that use an image skip the Playwright browsers cache (#343387), as the image holds the browsers. - The Electron stamp no longer hashes the deleted .npmrc. - The temporary validation runs every test job. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The checkout of a test job fetches and writes only what changed on main since the commit of its image, so the image doesn't need current source code: drop the build every 6 hours, and build the images only for pushes that change the inputs of the install, of the native builds or of the prepared Electron, whose stamp inputs join the path filter. The jobs no longer trust images of scheduled runs. When no input changes for 2 days, the images expire and the jobs install everything as before until the next one. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* ci: build workspace images of main for the PR test jobs Phase 1 of #345429: pr-workspace-images.yml builds a disk image of a ready workspace of main for each platform, when an input of the install, of the native builds or of the prepared Electron changes on main. Nothing uses the images yet: the test workflows adopt them one platform at a time in follow-up pull requests. setup-electron stamps the Electron that it prepares in .build/electron.stamp, and keeps an Electron prepared from the same inputs, as the images hold. Jobs that start from an empty workspace prepare Electron as before. Unlike in the pull request, image.ts has no `deps` command, as #346049 removed the install state that it compared: a job that uses an image runs its install steps on top of it, and pnpm installs only what changed. Co-authored-by: Alex Dima <alexdima@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: build the workspace images every day pr-workspace-images.yml also runs every day at 05:00 UTC, so that a recent image of main is always available: the images are kept for 2 days, and the pushes that change an input of the images can be days apart. image.ts trusts the images of the scheduled runs, which run on main like the pushes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: move the Windows workspace image scripts into a win32 folder vhd.ps1 and Vhd.cs only build and attach the Windows images, so they move to workspace_image/win32. The jobs download the scripts into the same folders, so that image.ts finds vhd.ps1 in win32 both in the repository and on the runners. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Alex Dima <alexdima@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Phase 2 of #345429, for Windows: the Windows test jobs of pull requests mount the workspace image that pr-workspace-images.yml builds on main, instead of creating an empty Dev Drive and restoring the node_modules and Playwright caches. With an image, the checkout keeps its untracked files and reuses its .git, and the usual install step runs on top of its node_modules, so pnpm installs only what changed since the commit of the image. Without an image, or on any error, the jobs set up as before. The Linux and macOS test jobs follow in their own pull requests. The smoke test shard with the WSL suite, on the 1ES pool, runs none of the image steps and keeps today's setup. Pull requests to release branches look for images of their own branch, find none, and set up as before. The windows-tests call in pr.yml grants actions: read, to find and download the images. The restore step passes --timeout-seconds 240, within the 5 minutes of the step, so that image.ts gives up and cleans up before the runner stops it. Unlike in the pull request, the jobs have no step that compares the dependencies with the image, as image.ts has no `deps` command since image.ts expects, as pr-workspace-images.yml does, which a test keeps equal. Co-authored-by: Alex Dima <alexdima@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Phase 2 of #345429, for Linux, after #346270 for Windows: the Linux test jobs of pull requests mount the workspace image that pr-workspace-images.yml builds on main, an ext4 volume on a loop device at the parent of the workspace, instead of restoring the node_modules and Playwright caches. With an image, the checkout keeps its untracked files and reuses its .git, and the usual install steps (the build root, then the root with setup-env.sh) run on top of its node_modules, so pnpm installs only what changed since the commit of the image, and setup-env.sh reuses the sysroots and compilers in its .build folder. The system packages are not in the image, and are installed as before. Without an image, or on any error, the jobs set up as before. The jobs download the scripts of the image exactly as the job that builds the Linux image does, which the test of the Windows jobs now also checks for the Linux jobs. Pull requests to release branches look for images of their own branch, find none, and set up as before. The linux-tests call in pr.yml grants actions: read, to find and download the images. The restore step passes --timeout-seconds 240, within the 5 minutes of the step, so that image.ts gives up, unmounts the image and detaches the loop device before the runner stops it. Co-authored-by: Alex Dima <alexdima@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

vs-code-engineering Bot assigned Alexandru Dima (alexdima) Nov 1, 2026

Ladislau Szomoru (lszomoru) force-pushed the lszomoru/pnpm12-migration branch 2 times, most recently from 5a984a8 to 97bf70c Compare November 2, 2026 07:29

Base automatically changed from lszomoru/pnpm12-migration to main November 2, 2026 08:04

Alexandru Dima (alexdima) force-pushed the agents/workspace-images branch from c83716a to 56f3d70 Compare November 2, 2026 11:45

Copilot AI balanced review requested due to automatic review settings November 2, 2026 11:45

Copilot started reviewing on behalf of Alexandru Dima (alexdima) November 2, 2026 11:45

View session

Copilot AI reviewed Nov 2, 2026

View reviewed changes

Comment thread .github/actions/setup-electron/action.yml Resolved

Comment thread .github/workflows/pr-linux-test.yml Resolved

Comment thread .github/workflows/pr-workspace-images.yml Resolved

Comment thread .github/workflows/workspace_image/image.ts Resolved

Alexandru Dima (alexdima) and others added 2 commits November 4, 2026 17:16

Ladislau Szomoru (lszomoru) force-pushed the agents/workspace-images branch from 9db8f88 to 7f92fbf Compare November 4, 2026 16:17

Ladislau Szomoru (lszomoru) mentioned this pull request Nov 5, 2026

ci: build workspace images of main for the PR test jobs #346211

Merged

This was referenced Nov 5, 2026

ci: give the workspace image restore a deadline, and clean up after it on Windows #346254

Merged

ci: start the Windows PR test jobs from the workspace images #346270

Merged

Ladislau Szomoru (lszomoru) mentioned this pull request Nov 5, 2026

ci: start the Linux PR test jobs from the workspace images #346362

Merged

This was referenced Nov 10, 2026

Engineering - start more pull request jobs from the workspace images #347275

Merged

Engineering - start the Windows chat-lib job from the workspace image #347287

Merged

Engineering - start the Component Fixtures jobs from the workspace images #347300

Merged

Navigation Menu

Repository navigation

ci: start the PR test jobs from workspace images of main - #345429

ci: start the PR test jobs from workspace images of main#345429
Alexandru Dima (alexdima) wants to merge 3 commits into
mainfrom
agents/workspace-images

Alexandru Dima (alexdima) commented Nov 1, 2026 •

edited by lszomoru

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Nov 2, 2026 •

edited

Loading

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Alexandru Dima (alexdima) commented Nov 1, 2026 • edited by lszomoru Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this changes

Results

Follow-ups

How to test

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Alexandru Dima (alexdima) commented Nov 1, 2026 •

edited by lszomoru

Loading

github-actions Bot commented Nov 2, 2026 •

edited

Loading