Skip to content

CI.md

Latest commit

 

History

History
397 lines (307 loc) · 25.9 KB

CI.md

File metadata and controls

397 lines (307 loc) · 25.9 KB
Raw

CI

This document describes how OpenShell's continuous integration works for pull requests, with a focus on what contributors need to do to get their PR tested.

For local test commands see TESTING.md. For PR conventions see CONTRIBUTING.md.

Overview

PR CI that runs on NVIDIA self-hosted runners uses NVIDIA's copy-pr-bot. The bot mirrors trusted PR commits to internal pull-request/<N> branches in this repository. The gated workflows trigger on pushes to those branches, not on the original PR.

Branch Checks run automatically after copy-pr-bot mirrors the PR. Required CI Gates posts PR-head statuses that verify the mirror exists, is current, and ran the expected push-based workflows. E2E suites are opt-in because they are more expensive and publish temporary images.

Merge queue validation is a second integration gate for main. After a PR has passed the required PR-head statuses, a maintainer adds it to the merge queue. GitHub creates a temporary merge-group branch that combines the latest main, the queued PR, and any earlier queued PRs. The same required OpenShell / ... status contexts are then published against the merge-group SHA before GitHub merges it.

Windows PR checks are opt-in: add test:windows, then select Re-run all jobs on the current Windows MSVC run. Subsequent mirrored commits run them automatically. Windows checks are not required for merging and do not run in merge queues. Main and manual runs also build release binaries, with continue-on-error: true so Windows failures do not fail the workflow.

Three opt-in labels enable the long-running E2E suites:

  • test:e2e runs the Docker, rootless Podman, Kubernetes, and VM E2E suites with both managed and standalone compute drivers in Branch E2E Checks
  • test:e2e-gpu runs GPU E2E in Branch E2E Checks
  • test:e2e-kubernetes runs Kubernetes E2E with the HA Helm overlay (replicaCount: 2 and bundled PostgreSQL) and the credential-driver suite (Kubernetes Secrets plus Vault) in Branch E2E Checks

When multiple labels are present, Branch E2E Checks builds each generic multi-architecture artifact set once and fans out enabled suites in parallel. Runtime-specific reusable workflows define the Docker, Podman, VM, and Kubernetes lanes. Composite actions own the replaceable Podman, KVM, kind, and mise setup. Each lane depends only on the artifact categories it consumes: VM does not wait for container-driver artifacts or supervisor images, and GPU does not wait for the gateway image. Docker, Podman, GPU, Rust, Python, MCP, and VM E2E reuse matching prebuilt gateway and CLI binaries instead of compiling debug binaries in test jobs. Standalone-driver lanes additionally reuse driver-free gateway and compute-driver artifacts. Kubernetes managed-driver lanes consume published gateway and supervisor images, while the standalone-driver lane composes its gateway image from prebuilt binaries. The OpenShell / E2E and OpenShell / GPU E2E required statuses are evaluated from separate suite result jobs inside that workflow. test:e2e-kubernetes is optional while Kubernetes HA and credential-driver behavior are under active iteration: failures are visible in the workflow run but do not publish a required CI gate status.

The GitHub ruleset should require the OpenShell / ... statuses published by Required CI Gates plus the direct OpenShell / Trivy Changes result, not the push-triggered workflow jobs themselves.

Informational security reports

Security workflow compute runs directly on GitHub-hosted runners instead of NVIDIA self-hosted runners. The PR-oriented workflows receive no secrets and run on fork pull requests without waiting for copy-pr-bot. Codex Security release qualification is the exception: it runs only for maintainer-created pre-release tags and receives a scoped NVIDIA Inference API key for the scan step. Scanner jobs request security-events: write to publish SARIF to Code Scanning. GitHub permits Code Scanning uploads from pull_request runs even when fork and Dependabot contexts receive a read-only GITHUB_TOKEN, so those scanners upload results directly:

  • Workflow Security Reports runs Actionlint and Zizmor. Actionlint reports workflow syntax and expression findings. Zizmor reports only High severity, its maximum level. Nix provides both scanners. They publish SARIF to Code Scanning and retain report artifacts.
  • Dependency Review compares the base and head dependency graphs and reports newly introduced vulnerabilities with a High-or-higher policy. It runs in warn-only mode. A preflight turns an unavailable GitHub Dependency Graph into a warning, so the workflow remains neutral until the repository feature is available.
  • CodeQL analyzes product Rust code, examples, and the Go, Python, and TypeScript SDKs. Rust cfg(test) blocks, Rust integration-test targets, and E2E test code are excluded. It runs nightly on main, remains manually dispatchable for diagnostics, uploads results to Code Scanning, and retains workflow artifacts.
  • Codex Security Release Qualification analyzes each vX.Y.Z-pre.N candidate against the previous stable release. Every candidate in a release train therefore rescans the cumulative stable-to-candidate diff. It calls https://inference-api.nvidia.com/v1 with openai/openai/gpt-5.6-sol at medium reasoning effort. The CODEX_SECURITY_API_KEY secret must contain an API key authorized for that NVIDIA endpoint. Results are uploaded against the candidate commit on main under a category shared by the train, for example codex-security/v0.1.1, so later candidates replace earlier analyses. The slash-qualified NVIDIA model identifier prevents Codex Security 0.1.24 from enforcing --max-cost, so the workflow relies on its timeout, serialized concurrency, and the inference account's spend controls. Raw reports are not retained as workflow artifacts. Pre-release creation and stable-promotion enforcement remain part of RFC 0014 and are not implemented by this workflow.

Findings do not fail these workflows. Tool startup, configuration, build, and analysis failures still fail so a broken scanner cannot appear healthy. The PR-oriented reports also run on merge groups; CodeQL is not a PR or merge-queue check. None of these reports are required statuses or gate merges.

Run the workflow-definition scanners locally with:

nix develop --command actionlint -shellcheck= -pyflakes=
nix develop --command zizmor --offline --persona=regular --min-severity=high --no-exit-codes .

Run the security scans together

Security Scan (.github/workflows/security-scan.yml) calls Codex Security, CodeQL, Trivy, Cargo Deny, and Workflow Security Reports (Actionlint/Zizmor) in parallel. Run it manually from Actions or call it from another workflow. All children scan candidate_ref, which must be an existing vX.Y.Z-pre.N tag because Codex qualifies pre-releases. Codex compares it with the previous stable; the other scanners analyze the candidate snapshot. Cargo Deny uses its existing NVIDIA self-hosted runner and CI container.

gh workflow run security-scan.yml --ref main \
  -f candidate_ref=v0.1.1-pre.1 \
  -F allow-high-critical=true

To integrate it into a larger workflow, run it after the job that pushes the candidate tag and publishes the artifacts. This example assumes an existing build job with outputs named candidate_tag, gateway_image, sandbox_image, and chart_ref; adapt those names to your workflow:

jobs:
  security:
    needs: build
    uses: ./.github/workflows/security-scan.yml
    permissions:
      actions: read
      contents: read
      packages: read
      security-events: write
    with:
      candidate_ref: ${{ needs.build.outputs.candidate_tag }}
      stable_ref: v0.1.0 # Optional; otherwise resolved automatically.
      images: |
        ${{ needs.build.outputs.gateway_image }}
        ${{ needs.build.outputs.sandbox_image }}
      charts: ${{ needs.build.outputs.chart_ref }}
      allow-high-critical: false
    secrets:
      CODEX_SECURITY_API_KEY: ${{ secrets.CODEX_SECURITY_API_KEY }}
      CACHIX_AUTH_TOKEN: ${{ secrets.CACHIX_AUTH_TOKEN }}

Set needs: security on a downstream promotion job to require successful scans.

CODEX_SECURITY_API_KEY is required; CACHIX_AUTH_TOKEN is optional. The parent publishes SARIF, including Codex results for manual parent runs. Codex keeps its release-train category on main; CodeQL, Trivy, and workflow reports publish against the candidate tag and commit. Existing standalone triggers stay active.

The parent fails on HIGH/CRITICAL findings from Codex, CodeQL, Trivy, and Zizmor. Set allow-high-critical: true to report those findings without failing; it defaults to false. Scanner setup, execution, and report publication errors still fail. Reports are published before the finding threshold is enforced.

Codex uses each finding's severity; CodeQL uses the rule's security score (at least 7.0); Trivy uses HIGH,CRITICAL, including vulnerabilities without an upstream fix; Zizmor uses its High severity. Actionlint remains informational. Existing scanner exceptions still apply.

Cargo Deny runs only cargo deny check advisories in the parent. It keeps its native failure behavior and configured exceptions, regardless of allow-high-critical; it has no HIGH/CRITICAL filter. Its standalone runs still check all dependency policies.

Codex scans the cumulative diff from stable_ref to candidate_ref. Both inputs are tag names, not arbitrary commit SHAs. Omit stable_ref to resolve the previous stable automatically; set allow_full_bootstrap: true to permit a full scan when no previous stable exists.

Trivy accepts optional images and charts, one reference per line. Images can use tags or digests, such as ghcr.io/nvidia/openshell/gateway:0.1.1-pre.1 or ghcr.io/nvidia/openshell/gateway@sha256:<digest>. Charts use versioned OCI references, such as oci://ghcr.io/nvidia/openshell/helm-chart:0.1.1-pre.1. Artifacts must already be published and accessible to the workflow. The caller selects artifacts matching the candidate; the workflow does not verify that association. Without artifact inputs, Trivy scans only the candidate's deployment configuration. Dependency Review and Trivy Changes keep their separate comparison workflows.

Artifact scanning

Trivy Scan is a self-contained workflow_dispatch/workflow_call step. It always scans deployment configuration and optionally scans supplied OCI image and chart references. Security Scan calls it alongside the other independent scanners; release workflows can also call it directly.

The scan job checks static deployment files (including Dockerfiles) once, both local charts with default values, the OpenShell HELM_PROFILES selected in tasks/scripts/trivy-scan.sh, requested packaged charts, and both Linux architectures of each requested image. The profile list includes development and E2E overlays for regression coverage, but excludes values-spire-stack.yaml, which belongs to the external SPIRE chart. A new OpenShell values fixture needs an explicit entry in this list to receive Trivy coverage.

Findings are informational by default, but scanner failures still fail the job and fail-on-findings enables enforcement. Reports retain every severity; the gate uses HIGH,CRITICAL by default.

All detailed JSON reports are retained in one artifact. The workflow summary and Code Scanning configuration report deduplicate the same rule, target, resource, and message across profiles, listing the affected profiles in each finding. Images and packaged charts retain separate analyses, keyed by full reference and platform so different registries or versions cannot overwrite each other. SARIF is generated only for publication. The upload job publishes only this consolidated configuration SARIF and the artifact analyses, in batches of at most 20 runs to respect GitHub's per-file limit. It also publishes complete reports when fail-on-findings makes the scan job fail; incomplete scans are retained as artifacts but are not published to Code Scanning.

.trivyignore.yaml is reserved for false positives. Every entry must use at least one **/<concrete-basename> path; yq and jq validate this structure before scanning.

Run the scanner locally with:

nix develop --command tasks/scripts/trivy-scan.sh config
nix develop --command tasks/scripts/trivy-scan.sh images ghcr.io/nvidia/openshell/gateway:dev
nix develop --command tasks/scripts/trivy-scan.sh gate
nix develop --command tasks/scripts/trivy-scan.sh prepare-sarif

Use a fresh TRIVY_REPORT_DIR for each scan session. prepare-sarif creates code-scanning/uploads/<batch>/; only these directories are intended for upload.

Pull-request change gate

Trivy Changes scans the base and candidate when a pull request changes deployment configuration or scanner inputs; merge groups and manual runs always scan. It fails only for new HIGH or CRITICAL misconfigurations and retains both report sets.

For pull requests, change detection compares the exact tested merge commit with its first parent. Both detection and scanning use that same pair. This excludes unrelated changes on main even when the event carries an older base SHA or a job is rerun. Merge groups use the event's base SHA; manual runs use the supplied base and head. A missing revision or invalid PR merge checkout fails the gate.

The candidate ignore file is validated, but the baseline policy applies to both scans so a change cannot exempt its own finding. Existing profiles are compared independently using the detailed reports, before any presentation deduplication; a new profile reuses the maximum known occurrence count. A selected fixture missing from the candidate fails the scan; a new fixture absent from the base has no baseline profile. Invalid Trivy reports fail closed. Image package and operating-system CVEs remain the responsibility of the standalone scan.

Commit signing

copy-pr-bot decides whether to mirror a PR automatically based on whether the author is trusted. For org members and collaborators, "trusted" means all commits in the PR are cryptographically signed. Unsigned commits, even from an org member, force the bot to wait for a maintainer's /ok to test <SHA>.

DCO sign-off (-s / Signed-off-by) is a separate requirement and does not count as commit signing. Dependabot-authored dependency update PRs are allowlisted in DCO Assistant because the bot cannot sign commits.

One-time setup with an SSH key

If you already use an SSH key for git push, you can reuse it as a signing key. (You can also generate a separate one - GitHub allows the same SSH key as both auth and signing.)

  1. Generate a key (skip if reusing your existing SSH key):

    ssh-keygen -t ed25519 -C "you@example.com" -f ~/.ssh/id_ed25519_signing

  2. Add the public key at https://github.com/settings/keys using New SSH key, and set Key type: Signing Key (not Authentication). Signing keys are managed separately from authentication keys, even when they reuse the same key material - you have to add the entry once for each role.

  3. Configure git globally:

    git config --global gpg.format ssh
git config --global user.signingkey ~/.ssh/id_ed25519_signing.pub
git config --global commit.gpgsign true
git config --global tag.gpgsign true

  4. Verify on a test commit:

    git commit --allow-empty -s -m "test: signing"

    Push the branch and confirm GitHub shows the commit as Verified.

Pull request flows

Internal contributor PR

Prerequisites:

  • Org member or collaborator on the repo.
  • All commits cryptographically signed (see Commit signing).
  • All commits include a DCO sign-off (git commit -s).

Flow:

  1. Open the PR. copy-pr-bot mirrors it to pull-request/<N> automatically.
  2. The mirror push runs Branch Checks automatically. Required CI Gates keeps the PR blocked until the mirror exists, matches the PR head SHA, and the required push-based workflow succeeds. The first Branch E2E Checks run only resolves metadata and skips expensive jobs unless an E2E label is already set.
  3. A maintainer applies test:e2e, test:e2e-gpu, and/or test:e2e-kubernetes. E2E Label Help posts a comment with a link to the existing gated workflow run.
  4. The maintainer opens that link and clicks Re-run all jobs. This time pr_metadata sees the label and the build/E2E jobs run.
  5. When the run finishes, the matching OpenShell / ... gate status flips to green automatically.
  6. New commits push to the mirror automatically and re-trigger Branch Checks plus any labeled E2E jobs in Branch E2E Checks.
  7. When the PR is ready to merge, use Add to merge queue instead of merging directly. The queue validates the final integration state before updating main.

Forked PR

Prerequisites:

  • DCO sign-off (git commit -s) on every commit. Commit signing is not required for forks - copy-pr-bot trusts forks based on maintainer review, not signing.
  • A maintainer must vouch you. See the Vouch System.

Flow:

  1. Open the PR. The vouch check confirms you are vouched (otherwise the PR is auto-closed).
  2. copy-pr-bot does not mirror forks automatically. A maintainer reviews the diff and comments /ok to test <SHA> with your latest commit SHA.
  3. After /ok to test, copy-pr-bot mirrors to pull-request/<N>. From here the flow is identical to internal PRs: Required CI Gates verifies the mirror and required push workflows, and maintainers apply the E2E label when the extra suites are needed.
  4. When the PR is ready to merge, maintainers add it to the merge queue so the queued integration state is tested before it reaches main.

Important: every new commit you push requires another /ok to test <new-SHA> from a maintainer before push-based CI will run on it. If a label is applied while the mirror is stale, E2E Label Help will post a comment explaining what's needed.

Merge queue

GitHub merge queue is required for main. Repository administrators must enable Require merge queue in the branch ruleset for main and keep these required status contexts aligned with the PR gates:

  • OpenShell / Branch Checks
  • OpenShell / E2E
  • OpenShell / GPU E2E
  • OpenShell / Helm Lint
  • OpenShell / Trivy Changes

Required CI Gates publishes the stable statuses for mirror-based workflows. Trivy Changes runs directly on pull requests and merge groups and publishes its own stable result status.

Merge-group runs use the merge_group event. The event is distinct from pull_request and push, and GitHub will not report required checks for queued PRs unless the workflows include it. In this repository:

  • Branch Checks runs the standard non-E2E gates on the merge-group SHA.
  • Branch E2E Checks runs core E2E and GPU E2E for merge groups. Kubernetes HA E2E remains optional and label-driven on PRs.
  • Helm Lint runs for merge groups without the PR diff optimization, because the merge-group branch is the final integration state.
  • Trivy Changes compares the merge-group configuration with its base and rejects new High or Critical findings.
  • Required CI Gates posts the same OpenShell / ... statuses to the merge-group SHA and does not require a pull-request/<N> mirror for merge-group events.

Maintainers should add ready PRs to the queue rather than pressing a direct merge button. GitHub removes a PR from the queue if the merge-group checks fail or time out.

copy-pr-bot

copy-pr-bot is a GitHub App maintained by NVIDIA that solves a specific GitHub Actions security problem: by default, pull_request-triggered workflows on a self-hosted runner can run an arbitrary contributor's code on hardware the project owns. For projects that need self-hosted runners (GPU access, ARM hardware, on-prem secrets), GitHub's recommended pattern is to never trigger workflows directly from external pull_request events.

copy-pr-bot enforces that pattern. When a PR is opened against this repository, the bot evaluates whether the change is trusted - by default, only commits authored by org members and signed with a verified key are trusted, and forks always need an explicit per-SHA approval. Once a change passes that check, the bot mirrors the PR head into a branch named pull-request/<N> inside this repository. Our self-hosted workflows then trigger on push to those mirror branches, never on the original pull_request event.

The user-visible consequences inside this repo:

  • A PR cannot run E2E until copy-pr-bot has mirrored it. For trusted authors this happens within seconds of opening the PR; for forked PRs it requires a maintainer to comment /ok to test <SHA>.
  • New commits to a fork need a fresh /ok to test <new-SHA> before the mirror updates.
  • The pull-request/<N> branches are not for humans to push to - they are managed by the bot.

The bot's full administrator documentation is internal to NVIDIA. The only command contributors may see in PR comments is /ok to test <SHA>, used by maintainers to approve a specific commit on a forked PR for testing.

Workflow files

File Role
.github/workflows/branch-checks.yml Required non-E2E checks. Triggers on push: pull-request/[0-9]+ for PR mirrors and merge_group for queued merges.
.github/workflows/branch-e2e.yml Standard, GPU, Kubernetes HA, and Kubernetes credential-driver E2E. PR mirror pushes use test:e2e, test:e2e-gpu, and test:e2e-kubernetes labels; merge groups run core and GPU E2E.
.github/workflows/build-{cli,gateway,sandbox}-binaries.yml Independent target matrices used by branch and release workflows without creating skipped jobs.
.github/workflows/package-release-binaries.yml Packages raw build artifacts into release tarballs without rebuilding them.
.github/workflows/e2e-docker-test.yml, e2e-podman-test.yml, e2e-vm-test.yml, e2e-kubernetes-test.yml Reusable runtime lanes called directly by branch and release workflows. Callers select suites and declare only the artifacts each runtime consumes.
.github/actions/setup-e2e-* Shared artifact, Podman, KVM, and kind setup used by the runtime lanes.
.github/workflows/helm-lint.yml Helm chart validation. PR mirror pushes skip lint jobs unless Helm inputs changed; merge groups always validate Helm because they represent the final integration state.
.github/actions/setup-nix/action.yml Installs Nix and configures the OpenShell Cachix cache, using read-only cache access when no authentication token is available.
.github/actions/pr-gate/action.yml Composite action that resolves PR metadata and verifies the required label is set for PR mirror pushes. Non-push events are allowed through.
.github/actions/pr-merge-base/action.yml Composite action that resolves and fetches the merge-base commit for pull-request/<N> push workflows.
.github/workflows/required-ci-gates.yml Posts required PR-head and merge-group statuses for gated CI workflows. This is what branch protection and merge queue should require.
.github/workflows/e2e-label-help.yml When a test:e2e* label is applied, posts a PR comment telling the maintainer the next manual step (re-run an existing workflow run, or /ok to test <SHA> to refresh the mirror).
.github/workflows/workflow-security.yml Runs informational Actionlint and High-severity Zizmor reports on GitHub-hosted runners.
.github/workflows/dependency-review.yml Reports dependency changes when GitHub Dependency Graph is available; otherwise publishes a neutral warning.
.github/workflows/codeql.yml Runs nightly informational CodeQL analysis on main for Rust and the Go, Python, and TypeScript SDKs and retains SARIF artifacts.
.github/workflows/codex-security.yml Scans the cumulative diff from the previous stable release to each pre-release candidate and publishes train-scoped SARIF on main.
.github/workflows/trivy-changes.yml Blocks pull requests and merge groups that introduce new High or Critical Helm or Dockerfile misconfigurations.
.github/workflows/trivy-scan.yml Manual or reusable scan of supplied OCI image/chart references and deployment configuration. Findings are informational by default and can be configured to fail the workflow.

Release workflows

These workflows run after merge to publish dev/tagged artifacts and verify them. They are not PR-gated.

File Role
.github/workflows/release-dev.yml Publishes the rolling dev build on every push to main. Builds gateway/supervisor images and binaries, packages, wheels, and pushes the Helm chart as oci://ghcr.io/nvidia/openshell/helm-chart:0.0.0-dev (plus an immutable 0.0.0-dev.<sha> pin). Also dispatchable manually.
.github/workflows/release-tag.yml Publishes a tagged stable release. Its automatic tag trigger excludes -pre.*; manual dispatch remains maintainer-controlled.
.github/workflows/release-canary.yml Smoke-tests published artifacts on macos, ubuntu, fedora, and kubernetes (kind + Helm) runners. Triggers automatically when Release Dev succeeds, and via workflow_dispatch on any branch (gh workflow run release-canary.yml --ref <branch>). The kubernetes job pins to 0.0.0-dev artifacts; the other jobs install the latest tagged release via install.sh. See the test-release-canary skill for the manual-dispatch playbook and local kind reproduction.

Required status contexts

Require these statuses in the branch ruleset for PR and merge-queue CI:

  • OpenShell / Branch Checks
  • OpenShell / E2E
  • OpenShell / GPU E2E
  • OpenShell / Helm Lint
  • OpenShell / Trivy Changes

For mirror-based workflows, require the statuses published by Required CI Gates, not their underlying jobs. OpenShell / Trivy Changes is the stable result job of the direct pull-request workflow. Together these contexts prove the expected checks completed for the commit GitHub is about to merge.

Do not add the informational Actionlint, Zizmor, Dependency Review, or CodeQL jobs to the required status list while they remain in observation mode.