Skip to content

unified base-image workflow with encrypt-at-dispatch credentials#681

Open
Yogeshwar-Pawade wants to merge 4 commits into
kairos-initfrom
feat/unified-base-images
Open

unified base-image workflow with encrypt-at-dispatch credentials#681
Yogeshwar-Pawade wants to merge 4 commits into
kairos-initfrom
feat/unified-base-images

Conversation

@Yogeshwar-Pawade

@Yogeshwar-Pawade Yogeshwar-Pawade commented Jul 13, 2026

Copy link
Copy Markdown

Summary

Unified base-image workflow that produces every OS variant a release needs in a single workflow_dispatch, with credentials handled via age-encrypted inputs so they never appear in cleartext anywhere.

Full-house dispatch (fips=true + trusted_boot=true + all creds) produces 12 kairosified base images pushed to us-east1-docker.pkg.dev/…/edge/kairos-*:

Empirically verified end-to-end in run 29276504878 — 12/12 images built and pushed, credential audit clean.

What this PR does

1. Extends the existing kairos-init base-images.yaml

generate-matrix + kairosify jobs are preserved unchanged — same docker/bake-action + docker-bake-kairosify.hcl pipeline that already produced Ubuntu/OpenSUSE/SLEM/UKI images successfully on this branch.

Adds two new jobs:

  • generate-extensions-matrix — computes RHEL + Ubuntu FIPS matrix rows based on which credentials the dispatcher supplied and whether fips=true.
  • extensions — matrix-parallel builder that runs docker build per RHEL/Ubuntu-FIPS variant using BuildKit --secret mounts.

collect-outputs now aggregates tags from both matrices.

Also:

  • trusted_boot=true is now additive (adds Ubuntu 24 UKI on top of the standard 5) rather than replacing them.
  • New fips checkbox mirrors the trusted_boot checkbox — check it to expand the matrix with FIPS variants.

2. Encrypt-at-dispatch credential handling

Instead of raw rhel_subscription_password, users paste age-encrypted ciphertexts. Ciphertexts are useless without the private key, which lives only as the WORKFLOW_DECRYPT_KEY repo secret and is auto-masked by GitHub.

# One-time on the operator's laptop
brew install age  # or apt/dnf install age

# Every dispatch
git checkout feat/unified-base-images
./scripts/encrypt-creds.sh
# → silent prompts for RHSM user, RHSM pass, Ubuntu Pro token
# → prints 3 ciphertext blocks

# On GitHub: Run workflow → paste each block into the matching input

Plaintext credentials never leave the operator's laptop.

Test plan

  • shellcheck clean across all scripts
  • yaml.safe_load parses base-images.yaml
  • Encrypt/decrypt roundtrip verified locally with test values
  • Empty-cipher path → script exits 0 with "nothing to decrypt"
  • Missing DECRYPT_KEY path → hard error with actionable message
  • Full-house dispatch run 29276504878 — 12/12 images built and pushed
  • Log audit of run 29276504878 — no plaintext credential material anywhere

…u-FIPS

Extends the existing kairos-init base-images.yaml (kairosify on Ubuntu,
OpenSUSE, SLEM, plus Ubuntu 24.04 UKI) with two additional families:

  * RHEL 8/9 standard      (needs RHSM creds)
  * RHEL 8/9 FIPS          (needs RHSM creds + fips checkbox)
  * Ubuntu 20.04/22.04 FIPS (needs Ubuntu Pro token + fips checkbox)

Everything pushes to the same $registry_prefix, using the existing
US_EAST_JSON_KEY_B64 secret for docker login.

Workflow shape:
  generate-matrix           → kairosify matrix (unchanged)
  kairosify                 → unchanged
  generate-extensions-matrix → new; masks creds, emits RHEL + Ubuntu-FIPS rows
                              based on which creds are set + fips checkbox
  extensions                 → new; docker build + push per row
  collect-outputs            → aggregates tags from both matrices

New inputs (4):
  fips                          checkbox, mirrors trusted_boot
  rhel_subscription_username    blank → skip RHEL
  rhel_subscription_password    masked; blank → skip RHEL
  ubuntu_pro_token              masked; required when fips=true

Credential rules:
  fips=true without ubuntu_pro_token           → hard error (matrix job)
  RHEL rows requested without RHSM             → soft-warn + skip
  Ubuntu FIPS requested without Pro token      → hard error (matrix would emit)

Credential handling:
  ::add-mask:: on first read in both matrix and build jobs
  Written to $RUNNER_TEMP/creds with umask 077
  Passed to docker build via BuildKit --secret; never --build-arg
  Shredded in an if: always() cleanup step

RHEL Dockerfile ports (rhel-core-images/rhel-fips/Dockerfile.rhel{8,9}):
  * Added `# syntax=docker/dockerfile:1` — required for BuildKit --mount
  * Removed ARG USERNAME / ARG PASSWORD
  * Replaced the register RUN with `RUN --mount=type=secret,id=rhsm_*`
    reading from /run/secrets/rhsm_username, /run/secrets/rhsm_password
  * Appended `subscription-manager unregister` + entitlement-state cleanup
    on the FIPS variants (rhel-core already had unregister; FIPS didn't)

Verified before push:
  * shellcheck clean across all 3 scripts
  * yaml.safe_load parses the workflow; 5 jobs, 11 inputs
  * Embedded python matrix generators lex-check clean
  * 5-scenario smoke test of the extensions matrix — every scenario matches
    the expected row count and tag format:
       no creds                → 0 extension rows (kairosify still runs)
       RHSM only               → 2 rows (RHEL 8/9 std)
       fips + all creds        → 6 rows (RHEL std + RHEL FIPS + Ubuntu FIPS)
       fips w/o Ubuntu Pro     → exits 1 with an ::error::
       fips w/o RHSM           → 2 rows Ubuntu FIPS only
  * All 4 RHEL Dockerfiles verified: syntax directive present, no ARG
    USERNAME/PASSWORD, --mount=type=secret present, unregister present

Prior earthly-based history preserved at tag
  archive/feat-unified-base-images-earthly-attempt
The previous behavior — inherited from kairos-init — used trusted_boot as
an exclusive mode: checking the box made the matrix contain ONLY the
Ubuntu 24.04 UKI row and skip everything else. Surprising: run
29216634756 (feat/unified-base-images) produced a single image instead
of the release set + the UKI variant.

Make trusted_boot additive:
  - Standard 5 rows (Ubuntu 20/22/24, OpenSUSE 15.6, SLEM 5.4) always run
  - Ubuntu 24.04 UKI appended when trusted_boot=true
  - Custom base_os_image override remains exclusive (single row, honors
    the trusted_boot flag on that row)

Standard rows are now emitted with `trusted_boot: False` regardless of
the workflow input; only the UKI row gets `trusted_boot: True`. The
bake-action now reads TRUSTED_BOOT from ${{ matrix.trusted_boot }}
instead of ${{ github.event.inputs.trusted_boot }} — otherwise every
standard row would get UKI-ified when the checkbox is on.

Symmetric to how the FIPS checkbox works in the extensions job: check
it → get FIPS variants added on top of the standard set.

Verified locally against the updated matrix generator:
  defaults        → 5 rows, all tb=false
  trusted_boot    → 6 rows (5 std + 1 UKI, per-row tb flag correct)
  base_os_image   → 1 row (exclusive, honors tb flag)
Two fixes surfaced by run 29266431825:

1. **Ubuntu FIPS 22.04 build failure.** The Dockerfile
   ubuntu-fips/22.04/Dockerfile.ubuntu22.04-fips has bare-filename COPYs
   (`COPY fix.sh /tmp/fix.sh`, `COPY modules.fips /tmp/modules.fips`)
   which resolve against the docker build context root. My matrix
   generator set build_context to `ubuntu-fips` (parent), so
   `/fix.sh` didn't exist in the context and the build failed with
   `failed to calculate checksum of ref ...: "/fix.sh": not found`.

   On the kairos-init branch, the per-version build.sh uses `.` (the
   version subdir itself) as context. My matrix now matches that: both
   20.04 and 22.04 use `ubuntu-fips/<version>` as the context.

2. **RHSM username leaked in log env dump.** The mask step only ran
   ::add-mask:: for password/token. In the Materialize step's env
   header, GitHub Actions logged:

     RHSM_USER: <plaintext email>
     RHSM_PASS: ***
     UBUNTU_PRO_TOKEN: ***

   The username is the account holder's email — personal identifier
   that shouldn't sit in the log. Add it to the mask loop in both the
   Mask (validate) and Re-mask (extensions job) steps. Same reminder
   still applies: `::add-mask::` acts on future output only; the mask
   step runs first in each job so subsequent env dumps are covered.

Verified from run 29266431825 logs:
  - No --build-arg USERNAME/PASSWORD anywhere (all --mount=type=secret)
  - subscription-manager register lines show $(cat /run/secrets/...)
    literally — never the resolved values
  - `pro attach` never appears with a plaintext token
  - Docker login line shows `echo "***" | base64 -d | docker login`
  - The only leak was RHSM_USER in the env header — closed here.
… anywhere

The previous plaintext credential inputs (rhel_subscription_username,
rhel_subscription_password, ubuntu_pro_token) inherently leaked in the
workflow log's env dump — the `::add-mask::` step could not cover its own
env: block, and workflow_dispatch string inputs are not auto-masked by
GitHub. Run 29271050030 exposed the actual RHSM password and Ubuntu Pro
token in cleartext.

This commit replaces plaintext inputs with age-encrypted ciphertexts:

Repo-side:
  - team.age.pub — public key, committed and safe to publish
  - scripts/encrypt-creds.sh — user-facing helper (silent prompts, encrypts
    each of RHSM user/pass/Ubuntu Pro with age, base64-encodes to
    single-line output, prints copy-pasteable blocks). Never echoes
    plaintext, never writes plaintext to a file.
  - scripts/README.md — user docs (install age, run script, paste
    ciphertexts, threat model, rotation policy)
  - .github/scripts/decrypt-creds.sh — workflow-side, reads three cipher
    env vars + the masked private key, decrypts with age, calls
    ::add-mask:: on each plaintext BEFORE exporting to $GITHUB_ENV. Wipes
    the private key file on exit via shred.

Workflow-side (.github/workflows/base-images.yaml):
  - Three inputs renamed with _encrypted suffix; description points at
    scripts/encrypt-creds.sh
  - "Mask credentials" step removed from generate-extensions-matrix — no
    plaintext exists in this job, so nothing to mask
  - "Re-mask credentials" step removed from extensions job — same reason
  - "Install age" step added on the runner (Luet-BigRunner has no age
    binary pre-installed)
  - "Decrypt credentials" step added — env: block contains ciphertexts
    (harmless without the key) and DECRYPT_KEY from secrets.* (auto-masked
    by GitHub before any log line)
  - "Materialize credentials" step no longer declares creds in env: —
    they arrive via process env from the Decrypt step's $GITHUB_ENV
    exports, already masked
  - HAS_RHSM_USER / HAS_RHSM_PASS / HAS_UBUNTU_PRO env vars in the matrix
    job now proxy on ciphertext-presence instead of plaintext-presence

Required repo secret (must be added before the next dispatch):
  WORKFLOW_DECRYPT_KEY = the age private key corresponding to team.age.pub

Verified locally (age installed via brew):
  - encrypt-creds.sh prompts silently, produces single-line ciphertexts
    (~280-300 bytes each — well under GH's 65KB input limit)
  - shellcheck clean across encrypt-creds.sh, decrypt-creds.sh, and the
    existing scripts
  - yaml.safe_load parses the workflow; 5 jobs, 11 inputs
  - Roundtrip: encrypt("fake-user@example.com") → base64 → decrypt-creds.sh
    → GITHUB_ENV file contains "RHSM_USER=fake-user@example.com" ✓
  - Empty-cipher case: script exits 0 with "nothing to decrypt"
  - Missing DECRYPT_KEY case: script hard-errors with actionable message

@bulwark-sc-ent bulwark-sc-ent Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Zizmor found Critical or High severity GitHub Actions workflow security issues:

Summary

Severity Count
High 23
Total 23

Details

Grouped by audit rule and file. Line/column refer to the workflow or action YAML on the scanned branch.

dangerous-triggers — High

use of fundamentally insecure workflow trigger

File: .github/workflows/backport.yaml

Fix guidance: https://docs.zizmor.sh/audits/#dangerous-triggers

Locations:

  • Line 2–4 (cols 0–32) — pull_request_target is almost always used insecurely

unpinned-uses — High

unpinned action reference

File: .github/workflows/backport.yaml

Fix guidance: https://docs.zizmor.sh/audits/#unpinned-uses

Locations:

  • Line 19 (cols 14–53) — expression sorenlouv/backport-github-action@v9.5.1 — action is not pinned to a hash (required by blanket policy)

template-injection — High (13 similar finding(s))

code injection via template expansion

File: .github/workflows/base-images.yaml

Fix guidance: https://docs.zizmor.sh/audits/#template-injection

Locations:

  • Line 92–142 (cols 8–13) — this step
  • Line 99 (cols 31–64) — expression github.event.inputs.base_os_image — may expand into attacker-controllable code
  • Line 94 (cols 8–11) — this run block
  • Line 100 (cols 33–68) — expression github.event.inputs.registry_prefix — may expand into attacker-controllable code
  • Line 101 (cols 22–46) — expression github.event.inputs.arch — may expand into attacker-controllable code
  • Line 102 (cols 23–48) — expression github.event.inputs.model — may expand into attacker-controllable code
  • Line 103 (cols 32–66) — expression github.event.inputs.kairos_version — may expand into attacker-controllable code
  • Line 104 (cols 30–62) — expression github.event.inputs.trusted_boot — may expand into attacker-controllable code
  • …and 9 more location(s) in this file.

unpinned-uses — High (8 similar finding(s))

unpinned action reference

File: .github/workflows/base-images.yaml

Fix guidance: https://docs.zizmor.sh/audits/#unpinned-uses

Locations:

  • Line 153 (cols 14–33) — expression actions/checkout@v4 — action is not pinned to a hash (required by blanket policy)
  • Line 156 (cols 14–43) — expression docker/setup-buildx-action@v3 — action is not pinned to a hash (required by blanket policy)
  • Line 162 (cols 14–35) — expression docker/bake-action@v6 — action is not pinned to a hash (required by blanket policy)
  • Line 184 (cols 14–40) — expression actions/upload-artifact@v4 — action is not pinned to a hash (required by blanket policy)
  • Line 294 (cols 14–33) — expression actions/checkout@v4 — action is not pinned to a hash (required by blanket policy)
  • Line 297 (cols 14–43) — expression docker/setup-buildx-action@v3 — action is not pinned to a hash (required by blanket policy)
  • Line 353 (cols 14–40) — expression actions/upload-artifact@v4 — action is not pinned to a hash (required by blanket policy)
  • Line 365 (cols 14–42) — expression actions/download-artifact@v4 — action is not pinned to a hash (required by blanket policy)

Please review these findings before merging.

@bulwark-sc-ent bulwark-sc-ent Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Zizmor found Critical or High severity GitHub Actions workflow security issues:

Summary

Severity Count
High 23
Total 23

Details

Grouped by audit rule and file. Line/column refer to the workflow or action YAML on the scanned branch.

dangerous-triggers — High

use of fundamentally insecure workflow trigger

File: .github/workflows/backport.yaml

Fix guidance: https://docs.zizmor.sh/audits/#dangerous-triggers

Locations:

  • Line 2–4 (cols 0–32) — pull_request_target is almost always used insecurely

unpinned-uses — High

unpinned action reference

File: .github/workflows/backport.yaml

Fix guidance: https://docs.zizmor.sh/audits/#unpinned-uses

Locations:

  • Line 19 (cols 14–53) — expression sorenlouv/backport-github-action@v9.5.1 — action is not pinned to a hash (required by blanket policy)

template-injection — High (13 similar finding(s))

code injection via template expansion

File: .github/workflows/base-images.yaml

Fix guidance: https://docs.zizmor.sh/audits/#template-injection

Locations:

  • Line 92–142 (cols 8–13) — this step
  • Line 99 (cols 31–64) — expression github.event.inputs.base_os_image — may expand into attacker-controllable code
  • Line 94 (cols 8–11) — this run block
  • Line 100 (cols 33–68) — expression github.event.inputs.registry_prefix — may expand into attacker-controllable code
  • Line 101 (cols 22–46) — expression github.event.inputs.arch — may expand into attacker-controllable code
  • Line 102 (cols 23–48) — expression github.event.inputs.model — may expand into attacker-controllable code
  • Line 103 (cols 32–66) — expression github.event.inputs.kairos_version — may expand into attacker-controllable code
  • Line 104 (cols 30–62) — expression github.event.inputs.trusted_boot — may expand into attacker-controllable code
  • …and 9 more location(s) in this file.

unpinned-uses — High (8 similar finding(s))

unpinned action reference

File: .github/workflows/base-images.yaml

Fix guidance: https://docs.zizmor.sh/audits/#unpinned-uses

Locations:

  • Line 153 (cols 14–33) — expression actions/checkout@v4 — action is not pinned to a hash (required by blanket policy)
  • Line 156 (cols 14–43) — expression docker/setup-buildx-action@v3 — action is not pinned to a hash (required by blanket policy)
  • Line 162 (cols 14–35) — expression docker/bake-action@v6 — action is not pinned to a hash (required by blanket policy)
  • Line 184 (cols 14–40) — expression actions/upload-artifact@v4 — action is not pinned to a hash (required by blanket policy)
  • Line 294 (cols 14–33) — expression actions/checkout@v4 — action is not pinned to a hash (required by blanket policy)
  • Line 297 (cols 14–43) — expression docker/setup-buildx-action@v3 — action is not pinned to a hash (required by blanket policy)
  • Line 353 (cols 14–40) — expression actions/upload-artifact@v4 — action is not pinned to a hash (required by blanket policy)
  • Line 365 (cols 14–42) — expression actions/download-artifact@v4 — action is not pinned to a hash (required by blanket policy)

Please review these findings before merging.

@Yogeshwar-Pawade Yogeshwar-Pawade changed the title feat(ci): unified base-image workflow with encrypt-at-dispatch credentials PE-8787: unified base-image workflow with encrypt-at-dispatch credentials Jul 13, 2026
@Yogeshwar-Pawade Yogeshwar-Pawade changed the title PE-8787: unified base-image workflow with encrypt-at-dispatch credentials unified base-image workflow with encrypt-at-dispatch credentials Jul 13, 2026
Yogeshwar-Pawade added a commit that referenced this pull request Jul 14, 2026
Main's .github/workflows/base-images.yaml is a stub (echo "Hello").
The real kairosify workflow lives on the long-lived origin/kairos-init
branch; port the 3 files needed to dispatch base-image builds:

  .github/workflows/base-images.yaml         (kairosify matrix, 188 lines)
  docker-bake-kairosify.hcl                  (bake target)
  dockerfiles/kairosify/Dockerfile.kairosify (kairos-init + BASE_OS_IMAGE)

Defaults bumped to match PE-8787's target versions:
  kairos_init_image: v0.8.12   → v0.16.1
  kairos_version:    v4.0.1    → v4.1.2
  registry_prefix:   pe-8215/… → pe-8787/…

  docker-bake-kairosify.hcl KAIROS_INIT_IMAGE v0.8.12 → v0.16.1
  docker-bake-kairosify.hcl KAIROS_VERSION    v3.5.9  → v4.1.2

Covers kairosify targets only (Ubuntu 22/24, OpenSUSE 15.6, SLEM 5.4/5.5,
Ubuntu 24.04 UKI). RHEL / RHEL FIPS / Ubuntu FIPS are handled by PR #681
(feat/unified-base-images) — kept separate here to preserve review scope.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant