Skip to content

Let the benchmark signal go red without holding the deploy back - #1008

Merged
epompeii merged 1 commit into
develfrom
u/ep/benchmarks-non-gating
Aug 23, 2026
Merged

Let the benchmark signal go red without holding the deploy back#1008
epompeii merged 1 commit into
develfrom
u/ep/benchmarks-non-gating

Conversation

@epompeii

Copy link
Copy Markdown
Member

Problem

On the devel push run 32651179276, Test / Track Benchmarks / Track Benchmarks with Bencher failed on benchmark alerts (an accepted latency regression). That one failure cascaded all the way to the deploy:

  1. bencher run ... --error-on-alert exits non-zero, so the track_benchmarks job in .github/workflows/track_benchmarks.yml fails.
  2. That job is the only job in its workflow, so the reusable Track Benchmarks workflow concludes failure.
  3. test.yml calls it as the track_benchmarks job, so that caller job's result is failure, and the reusable Test workflow concludes failure.
  4. In ci.yml, deploy is needs: [lint, cli, test, build] with if: ${{ !failure() && !cancelled() && ... }}. With needs.test.result == 'failure', !failure() is false and Deploy is skipped.
  5. ci-success then reads needs.test.result != 'success' and exits 1, so CI Success is red too.

A benchmark alert is a signal about performance. It is not a statement about whether the build is deployable.

The change

Benchmark alerts inform, they never gate. One line, continue-on-error: true, on the track_benchmarks job in .github/workflows/track_benchmarks.yml.

Why there and not at the call site

continue-on-error is not among the keywords GitHub accepts on a job that calls a reusable workflow (jobs.<id>.uses supports name, uses, with, secrets, needs, if, permissions, strategy, and concurrency). The flag has to live on a real job inside the callee.

This is exactly how Lint / Cargo Deny Advisories and Lint / Zizmor Online already work: both carry continue-on-error: true at job level inside the reusable lint.yml, never at the ci.yml call site, and ci-success reads needs.lint.result and stays green when either of them fails. The same conclusion-laundering applies here, once per uses: hop.

Propagation after the change

  1. The track_benchmarks job fails on alerts and is rendered red on the run, so the signal stays visible. --error-on-alert is untouched, as is the --github-actions check that reports the alert.
  2. continue-on-error: true keeps that failure from failing the workflow, so Track Benchmarks concludes success.
  3. The track_benchmarks caller job in test.yml therefore succeeds. Nothing in test.yml has needs: [track_benchmarks], so no downstream job is skipped, and Test concludes success.
  4. needs.test.result == 'success' in ci.yml, so deploy runs and ci-success passes.

ci-success enumerates job results one by one, but only at the top level (lint, cli, runner, publish_runner_canary, test, build, docker, deploy, release). There is no per-benchmark entry to adjust, so the single line is enough.

track_pr_benchmarks.yml is left alone

The PR side cascade gates nothing. Track PR Benchmarks is a separate workflow_run workflow, no job needs it, it is not among ci-success's dependencies, and PR runs do not deploy. Its red X on an alert is the intended pull request signal, so it keeps --error-on-alert and stays gating-free by construction.

@epompeii
epompeii force-pushed the u/ep/benchmarks-non-gating branch from ef4b427 to 4db3e25 Compare August 23, 2026 16:59
@epompeii
epompeii marked this pull request as ready for review August 23, 2026 17:00
@epompeii
epompeii merged commit 37abefb into devel Aug 23, 2026
9 of 15 checks passed
@epompeii
epompeii deleted the u/ep/benchmarks-non-gating branch August 23, 2026 17:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant