Repository navigation
Conversation
The mise cache guard keys each cached mise step by the Ubuntu image its job runs on. Jobs built on one image fail with "GLIBC_2.38 not found" when they restore tools built on another. The guard found the image by looking for `ubuntu-NNNN` in the `runs-on` label. A RunsOn label names its image differently, as `image=ubuntu22-full-x64`. The heavy jobs are about to move to RunsOn. The guard would then report every one of them as having no image. So the guard now reads the `image=` segment of a RunsOn label. A RunsOn label without that segment uses the RunsOn stack's default image. The label does not show that image, so the mise cache must stay off for such a job.
CI on Blacksmith cost about $3,865 a month at October's pace. The budget for this repository and the others that share it is $1,000 a month. RunsOn starts a machine in our own AWS account for each job. In the runner trial in #1297, a week of these jobs cost $194 on RunsOn c8a with 8 vCPUs. The same jobs cost $843 on Blacksmith. They took 0.93 times as long on RunsOn. This moves every heavy job that ran on `blacksmith-16vcpu-ubuntu-2204`. A heavy job compiles Rust, builds a Docker image or runs a Postgres suite. Each one now runs on c8a with 8 vCPUs. Mutants runs on c8a with 16 vCPUs. It tests one mutant at a time, and in the trial 16 vCPUs ran it about 25% faster than 8. Every label also carries these settings: - `family=c8a+m8a`: m8a has the same CPU as c8a. RunsOn can use it when AWS has no spare c8a spot machines, before paying the on-demand price. - `image=ubuntu22-full-x64`: Ubuntu 22.04, as on the Blacksmith label. The mise cache prefixes stay `mise-v1-ubuntu-2204`. - `volume=80gb`: the default disk fills during a Rust build. - `extras=s3-cache`: the job cache comes from RunsOn's S3 bucket in our account. Fetching GitHub's cache from AWS is slow. The light jobs on that label stay on Blacksmith for now. They are change detection, the matrix setup, two bash-only checks, the summary job, and an image re-tag. They move to GitHub-hosted runners in a later change. actionlint does not check a label that contains an expression. So the RunsOn labels need no entry in `.github/actionlint.yaml`.
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Comment |
|
Contributor
Mutation testing (cargo-mutants,
|
Contributor
Author
First CI run on RunsOn: every moved job passedOn commit b21019d, all 22 jobs that ran with a RunsOn label passed. All 13 workflow runs passed. Two review workflows were skipped, as they are on every draft.
Still not run on RunsOn: Bench EQL, the macro expansion check, and the release jobs. None of them run on a pull request. |
A heavy job is one that runs too slowly on a GitHub-hosted standard runner. For the 4 vCPU jobs, the rule comes from the runner trial in #1297. A job is heavy when GitHub's runner took more than twice as long as Blacksmith. All four integration suites pass that rule. In the trial's warm run, GitHub's runner took these multiples of Blacksmith 4 vCPU's time: - Drizzle: 3.42 (db=postgres) and 3.68 (db=supabase) - prisma-next: 3.00 (db=postgres) and 2.68 (db=supabase) - Supabase: 2.96 - protect-ffi: 4.78 The warm run is the trial's like-for-like comparison, because every runner had a warm cache. On the trial's first run some caches were cold. They move to RunsOn with 8 vCPUs, like the other heavy jobs. In the trial's warm run, c8a with 8 vCPUs took 0.84 to 1.05 times Blacksmith 4 vCPU's time on these jobs. They keep Ubuntu 24.04 through `image=ubuntu24-full-x64`, so protect-ffi's `mise-v1-ubuntu-2404` cache prefix stays valid. udeps stays on Blacksmith. GitHub's runner took 1.17 times as long in the warm run.
Contributor
Author
Second CI run: Docker Hub refuses some image pulls on RunsOnBlocker. On commit 5da3bd9, three jobs failed while starting Postgres. Docker Hub answered
The other 25 RunsOn jobs passed. They include the five Drizzle, prisma-next, and Supabase integration jobs. All 28 jobs got a machine within 18 to 25 seconds. The trial in #1297 did not report this error. I cannot see from the logs whether RunsOn machines share an outbound address. Each machine reports Which jobs pull from Docker Hub
The Ways to clear it
Bench EQL and the macro expansion checkI started both by hand on this branch. Both got an m8a.2xlarge spot machine within 19 seconds, and both passed. Bench EQL took about 23 minutes. |
This was referenced Oct 28, 2026
Docker Hub refuses pulls without a login after too many from one address. RunsOn machines reach that limit. On this branch, Test EQL shards failed with `toomanyrequests` before any test ran, and failed again on a re-run. The protect-ffi suite failed the same way once. Every RunsOn job that starts or builds a container now logs in first, with the DOCKER_HUB_USERNAME and DOCKER_HUB_PASSWORD secrets: - A `run:` step pipes the password to `docker login --password-stdin`. It comes before Postgres starts in Test EQL, Bench EQL, and the four integration suites. In the postgres-eql image release, it comes before QEMU and Buildx, because both pull their own images from Docker Hub. - The Postgres service container in Test EQL's `rust-crates` job sets `credentials` from the same secrets. Both forms do nothing when either secret is empty. That is the case on a fork's pull request, and until the secrets exist. The step exits early, and the runner skips the service login when either value is empty. Pulls then stay anonymous, as before. `workflow-docker-hub-login.test.mjs` fails when a RunsOn job may pull an image without logging in first. Any step that starts or builds a container counts as a pull, whatever the registry. So the test cannot miss a Docker Hub image it does not recognise. It follows `mise run` into task bodies, `depends`, and nested `mise run` calls, and into local composite actions.
The previous commit gave Test EQL's Postgres service `username:` and
`password:` keys from the two Docker Hub secrets. The secrets do not
exist yet, so both values were empty. GitHub rejects an empty value for
either key before the runner starts: "The template is not valid ...
Unexpected value ''". The `rust-crates` job failed at "Set up job". A
fork's pull request would fail the same way.
`credentials` now takes one expression. With both secrets set, it builds
the username and password from them, through `toJSON`, so any character
is escaped. With either secret empty, it gives `{}`. The workflow schema
allows that, because neither key is required. The runner then skips the
login and pulls without one.
The test now rejects the key form and requires the `fromJSON('{}')`
fallback.
A 4 vCPU job counts as heavy when GitHub's standard runner takes more than twice as long as Blacksmith. The runner trial in #1297 did not time the FFI builds. So they ran on `ubuntu-24.04` three times, through `ffi-preflight.yml`. That workflow cannot publish. These are the medians: - linux-x64-gnu: 258 s on GitHub's runner, 125 s on Blacksmith (2.06). - linux-arm64-gnu: 288 s on GitHub's runner, 143 s on Blacksmith (2.01). Both are only just over the line. They move to RunsOn with 8 vCPUs and Ubuntu 24.04, as Blacksmith used. The runner comes from `runnerFor` in the matrix script, not from the workflow. A matrix value cannot read `${{ github.run_id }}`, so the script puts `GITHUB_RUN_ID` into the label. It throws when the run ID is missing, because a label without it would be shared by every run. Three Linux jobs stay as they are: - The musl build could not be timed. Its image pins `musl-dev=1.2.6-r2`. Alpine no longer serves that version, so the build fails on any runner. That also breaks the next FFI release. - The WASM and wrapper build needs every binary, so it never ran. - Miri and tests-rust took 1.35 and 1.36 times as long on GitHub's runner, so they are light.
On its first RunsOn run, the linux-arm64-gnu FFI build failed in its `apt-get install`: "Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 1720 (apt)". A RunsOn machine boots for each job, and apt was still running from boot. GitHub's and Blacksmith's images do not run apt at boot, so the step never met a held lock there. apt-get fails at once on a held lock unless told to wait. Both RunsOn jobs that install packages now pass `-o DPkg::Lock::Timeout=300`: the gnu FFI build's cross toolchain and the EQL docs build's Doxygen.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Before merging, add two repository secrets:
DOCKER_HUB_USERNAMEandDOCKER_HUB_PASSWORD. Docker Hub refuses pulls without a login after too many from one address, and RunsOn machines reach that limit. Until the secrets exist, the jobs pull without a login and can still fail on that limit.Summary
This moves the heavy CI jobs from Blacksmith to RunsOn. Blacksmith is the hosted runner service our CI uses today. RunsOn starts a fresh machine for each job in our own AWS account.
CI on Blacksmith cost about $3,865 a month at October's pace. The budget for this repository and the others that share it is $1,000 a month. In the runner trial in #1297, a week of these jobs cost $194 on RunsOn with 8 vCPUs. The same jobs cost $843 on Blacksmith. They took 0.93 times as long on RunsOn.
A heavy job needs more machine than a GitHub-hosted standard runner gives it. In this repository that means Rust compiles, Postgres suites, Docker image builds, and the integration suites.
Changes
Runner labels. Every heavy job on
blacksmith-16vcpu-ubuntu-2204now runs on this RunsOn label:family=c8a+m8a: AMD c8a machines. RunsOn can use m8a when AWS has no spare c8a spot machines. m8a has the same CPU.cpu=8: 8 vCPUs. Mutants usescpu=16.image=ubuntu22-full-x64: Ubuntu 22.04, the same as the Blacksmith label.volume=80gb: the default disk fills during a Rust build.extras=s3-cache: the job cache comes from RunsOn's S3 bucket in our account.16 vCPU jobs moved to 8 vCPUs (25 jobs):
16 vCPU job moved to 16 vCPUs: Mutants. It tests one mutant at a time. In the trial, 16 vCPUs ran it about 25% faster than 8.
4 vCPU jobs moved to 8 vCPUs: the four integration suites. A 4 vCPU job counts as heavy when GitHub's runner took more than twice as long as Blacksmith in #1297. They keep Ubuntu 24.04 with
image=ubuntu24-full-x64. These are the times from the trial's warm run, in minutes:4 vCPU jobs moved to 8 vCPUs: the gnu FFI release builds. #1297 did not time them. So #1438 ran them three times on
ubuntu-24.04, throughffi-preflight.yml. That workflow cannot publish. Times are in seconds:Both are just over the line. On RunsOn c8a with 8 vCPUs, they took 88 s and 101 s. Their runner comes from
runnerForinscripts/ffi-release-matrix.mjs. A matrix value cannot read${{ github.run_id }}, so the script putsGITHUB_RUN_IDinto the RunsOn label. The script throws if the run ID is missing.Jobs left on Blacksmith.
ci-required.:latest.apk add musl-dev=1.2.6-r2, and Alpine now serves only1.2.6-r3. The wrapper build waits for every binary, so it was skipped.Docker Hub login. Every RunsOn job that starts or builds a container now logs in to Docker Hub first:
Log in to Docker Hubstep pipes the password todocker login --password-stdin. It comes before Postgres starts in Test EQL (5 jobs), Bench EQL, and the four integration suites. In the postgres-eql image release, it comes before QEMU and Buildx, because both pull images from Docker Hub.rust-cratesjob setscredentialsto one expression. It reads both secrets, and gives{}when either is empty.username:andpassword:keys do not work here: GitHub rejects an empty value for either ("Unexpected value ''"), and the job fails at "Set up job". That happened on b870350.{}the runner pulls the service image without a login.apt lock. On its first RunsOn run, the linux-arm64-gnu build failed in
apt-get install: "Could not get lock /var/lib/dpkg/lock-frontend". A RunsOn machine boots for each job, and apt was still running from boot. The step had passed on every Blacksmith and GitHub-hosted run.Both RunsOn jobs that install packages now pass
-o DPkg::Lock::Timeout=300, so apt waits for the lock. They are the gnu FFI build and the EQL docs build.Tests and config:
scripts/__tests__/workflow-docker-hub-login.test.mjsis new. It fails when a RunsOn job may pull an image without logging in first, or when a login would fail on a fork's pull request. Any step that starts or builds a container counts as a pull, whatever the registry. It followsmise runinto task bodies,depends, and nested calls, and into local composite actions. It reads only literalruns-on=labels, so it does not see the FFI matrix. The gnu FFI builds pull no images.scripts/__tests__/ffi-release-matrix.test.mjspins the new labels, and the error for a missing run ID.scripts/__tests__/workflow-mise-cache-image.test.mjsnow reads theimage=part of a RunsOn label. This test makes each mise cache key name the job's Ubuntu image. Without the change, it reports every moved job as having no image..github/actionlint.yaml: comments only. actionlint does not check a label that contains an expression, so RunsOn labels need no entry.Verification
pnpm run test:scripts: 81 files passed. 1,429 tests passed and 3 were skipped.I removed one login step and the service
credentialsfrom copies of two workflows. The new test reported both, then passed once I restored them.I ran
main's copy of the mise cache test against these workflows. It failed on every moved job with a mise cache, so the test does read the new labels.e2e/tests/supply-chain.e2e.test.ts: 29 tests passed.actionlint 1.7.12 with shellcheck 0.11.0 on all 16 changed workflows, and again on the 7 that gained a login. It reported one warning, SC2034 in
release-golang.ymlat line 406.mainhas the same warning. It is in a job this PR does not change.biome checkon the changed test file: clean.CI on b21019d: all 22 moved jobs that run on a pull request passed on RunsOn. The results are in a comment below.
CI on 5da3bd9: three jobs failed because Docker Hub refused to pull an image without a login. They are Test EQL shards 1/4 and 4/4 and the protect-ffi suite. The shards failed again on a re-run, and protect-ffi passed. The other 25 RunsOn jobs passed. A comment below gives the details and three ways to clear it.
Bench EQL and the macro expansion check, started by hand on this branch: both ran on RunsOn and passed.
CI on 6e1db64, with no Docker Hub secrets set: all 16 workflow runs and all 28 RunsOn jobs passed. Each login step printed that pulls stay anonymous.
rust-cratespulledpostgres:17without a login.ffi-preflight.yml, started by hand on this branch twice. It cannot publish. Both gnu builds got RunsOn machines within 23 seconds. On the first run, linux-arm64-gnu hit the apt lock. On the second, after the fix, both passed. The musl build failed on Blacksmith both times, on theapkpin.Not verified:
Related
Review notes
envinruns-on. A repository variable would need a settings change.family=c8a+m8ais a prefix match in RunsOn. In the trial,family=c8galso gave c8gd and c8gn machines. If RunsOn hands out unexpected m8a variants,family=c8a.*+m8a.*would pin them.extras=s3-cache. These jobs restore no cache, becausescripts/lint-no-workflow-caching.mjsforbids cache steps there. On Blacksmith they already ran beside its transparent cache.apkpin breaks the next FFI release on any runner. This pull request does not fix it.timeout-minutesto every job, and another adds selector jobs. Their changes sit next to theseruns-onlines, so whichever merges second will need a small rebase.