unified base-image workflow with encrypt-at-dispatch credentials#681
unified base-image workflow with encrypt-at-dispatch credentials#681Yogeshwar-Pawade wants to merge 4 commits into
Conversation
…u-FIPS
Extends the existing kairos-init base-images.yaml (kairosify on Ubuntu,
OpenSUSE, SLEM, plus Ubuntu 24.04 UKI) with two additional families:
* RHEL 8/9 standard (needs RHSM creds)
* RHEL 8/9 FIPS (needs RHSM creds + fips checkbox)
* Ubuntu 20.04/22.04 FIPS (needs Ubuntu Pro token + fips checkbox)
Everything pushes to the same $registry_prefix, using the existing
US_EAST_JSON_KEY_B64 secret for docker login.
Workflow shape:
generate-matrix → kairosify matrix (unchanged)
kairosify → unchanged
generate-extensions-matrix → new; masks creds, emits RHEL + Ubuntu-FIPS rows
based on which creds are set + fips checkbox
extensions → new; docker build + push per row
collect-outputs → aggregates tags from both matrices
New inputs (4):
fips checkbox, mirrors trusted_boot
rhel_subscription_username blank → skip RHEL
rhel_subscription_password masked; blank → skip RHEL
ubuntu_pro_token masked; required when fips=true
Credential rules:
fips=true without ubuntu_pro_token → hard error (matrix job)
RHEL rows requested without RHSM → soft-warn + skip
Ubuntu FIPS requested without Pro token → hard error (matrix would emit)
Credential handling:
::add-mask:: on first read in both matrix and build jobs
Written to $RUNNER_TEMP/creds with umask 077
Passed to docker build via BuildKit --secret; never --build-arg
Shredded in an if: always() cleanup step
RHEL Dockerfile ports (rhel-core-images/rhel-fips/Dockerfile.rhel{8,9}):
* Added `# syntax=docker/dockerfile:1` — required for BuildKit --mount
* Removed ARG USERNAME / ARG PASSWORD
* Replaced the register RUN with `RUN --mount=type=secret,id=rhsm_*`
reading from /run/secrets/rhsm_username, /run/secrets/rhsm_password
* Appended `subscription-manager unregister` + entitlement-state cleanup
on the FIPS variants (rhel-core already had unregister; FIPS didn't)
Verified before push:
* shellcheck clean across all 3 scripts
* yaml.safe_load parses the workflow; 5 jobs, 11 inputs
* Embedded python matrix generators lex-check clean
* 5-scenario smoke test of the extensions matrix — every scenario matches
the expected row count and tag format:
no creds → 0 extension rows (kairosify still runs)
RHSM only → 2 rows (RHEL 8/9 std)
fips + all creds → 6 rows (RHEL std + RHEL FIPS + Ubuntu FIPS)
fips w/o Ubuntu Pro → exits 1 with an ::error::
fips w/o RHSM → 2 rows Ubuntu FIPS only
* All 4 RHEL Dockerfiles verified: syntax directive present, no ARG
USERNAME/PASSWORD, --mount=type=secret present, unregister present
Prior earthly-based history preserved at tag
archive/feat-unified-base-images-earthly-attempt
The previous behavior — inherited from kairos-init — used trusted_boot as
an exclusive mode: checking the box made the matrix contain ONLY the
Ubuntu 24.04 UKI row and skip everything else. Surprising: run
29216634756 (feat/unified-base-images) produced a single image instead
of the release set + the UKI variant.
Make trusted_boot additive:
- Standard 5 rows (Ubuntu 20/22/24, OpenSUSE 15.6, SLEM 5.4) always run
- Ubuntu 24.04 UKI appended when trusted_boot=true
- Custom base_os_image override remains exclusive (single row, honors
the trusted_boot flag on that row)
Standard rows are now emitted with `trusted_boot: False` regardless of
the workflow input; only the UKI row gets `trusted_boot: True`. The
bake-action now reads TRUSTED_BOOT from ${{ matrix.trusted_boot }}
instead of ${{ github.event.inputs.trusted_boot }} — otherwise every
standard row would get UKI-ified when the checkbox is on.
Symmetric to how the FIPS checkbox works in the extensions job: check
it → get FIPS variants added on top of the standard set.
Verified locally against the updated matrix generator:
defaults → 5 rows, all tb=false
trusted_boot → 6 rows (5 std + 1 UKI, per-row tb flag correct)
base_os_image → 1 row (exclusive, honors tb flag)
Two fixes surfaced by run 29266431825:
1. **Ubuntu FIPS 22.04 build failure.** The Dockerfile
ubuntu-fips/22.04/Dockerfile.ubuntu22.04-fips has bare-filename COPYs
(`COPY fix.sh /tmp/fix.sh`, `COPY modules.fips /tmp/modules.fips`)
which resolve against the docker build context root. My matrix
generator set build_context to `ubuntu-fips` (parent), so
`/fix.sh` didn't exist in the context and the build failed with
`failed to calculate checksum of ref ...: "/fix.sh": not found`.
On the kairos-init branch, the per-version build.sh uses `.` (the
version subdir itself) as context. My matrix now matches that: both
20.04 and 22.04 use `ubuntu-fips/<version>` as the context.
2. **RHSM username leaked in log env dump.** The mask step only ran
::add-mask:: for password/token. In the Materialize step's env
header, GitHub Actions logged:
RHSM_USER: <plaintext email>
RHSM_PASS: ***
UBUNTU_PRO_TOKEN: ***
The username is the account holder's email — personal identifier
that shouldn't sit in the log. Add it to the mask loop in both the
Mask (validate) and Re-mask (extensions job) steps. Same reminder
still applies: `::add-mask::` acts on future output only; the mask
step runs first in each job so subsequent env dumps are covered.
Verified from run 29266431825 logs:
- No --build-arg USERNAME/PASSWORD anywhere (all --mount=type=secret)
- subscription-manager register lines show $(cat /run/secrets/...)
literally — never the resolved values
- `pro attach` never appears with a plaintext token
- Docker login line shows `echo "***" | base64 -d | docker login`
- The only leak was RHSM_USER in the env header — closed here.
… anywhere
The previous plaintext credential inputs (rhel_subscription_username,
rhel_subscription_password, ubuntu_pro_token) inherently leaked in the
workflow log's env dump — the `::add-mask::` step could not cover its own
env: block, and workflow_dispatch string inputs are not auto-masked by
GitHub. Run 29271050030 exposed the actual RHSM password and Ubuntu Pro
token in cleartext.
This commit replaces plaintext inputs with age-encrypted ciphertexts:
Repo-side:
- team.age.pub — public key, committed and safe to publish
- scripts/encrypt-creds.sh — user-facing helper (silent prompts, encrypts
each of RHSM user/pass/Ubuntu Pro with age, base64-encodes to
single-line output, prints copy-pasteable blocks). Never echoes
plaintext, never writes plaintext to a file.
- scripts/README.md — user docs (install age, run script, paste
ciphertexts, threat model, rotation policy)
- .github/scripts/decrypt-creds.sh — workflow-side, reads three cipher
env vars + the masked private key, decrypts with age, calls
::add-mask:: on each plaintext BEFORE exporting to $GITHUB_ENV. Wipes
the private key file on exit via shred.
Workflow-side (.github/workflows/base-images.yaml):
- Three inputs renamed with _encrypted suffix; description points at
scripts/encrypt-creds.sh
- "Mask credentials" step removed from generate-extensions-matrix — no
plaintext exists in this job, so nothing to mask
- "Re-mask credentials" step removed from extensions job — same reason
- "Install age" step added on the runner (Luet-BigRunner has no age
binary pre-installed)
- "Decrypt credentials" step added — env: block contains ciphertexts
(harmless without the key) and DECRYPT_KEY from secrets.* (auto-masked
by GitHub before any log line)
- "Materialize credentials" step no longer declares creds in env: —
they arrive via process env from the Decrypt step's $GITHUB_ENV
exports, already masked
- HAS_RHSM_USER / HAS_RHSM_PASS / HAS_UBUNTU_PRO env vars in the matrix
job now proxy on ciphertext-presence instead of plaintext-presence
Required repo secret (must be added before the next dispatch):
WORKFLOW_DECRYPT_KEY = the age private key corresponding to team.age.pub
Verified locally (age installed via brew):
- encrypt-creds.sh prompts silently, produces single-line ciphertexts
(~280-300 bytes each — well under GH's 65KB input limit)
- shellcheck clean across encrypt-creds.sh, decrypt-creds.sh, and the
existing scripts
- yaml.safe_load parses the workflow; 5 jobs, 11 inputs
- Roundtrip: encrypt("fake-user@example.com") → base64 → decrypt-creds.sh
→ GITHUB_ENV file contains "RHSM_USER=fake-user@example.com" ✓
- Empty-cipher case: script exits 0 with "nothing to decrypt"
- Missing DECRYPT_KEY case: script hard-errors with actionable message
There was a problem hiding this comment.
Summary
| Severity | Count |
|---|---|
| High | 23 |
| Total | 23 |
Details
Grouped by audit rule and file. Line/column refer to the workflow or action YAML on the scanned branch.
dangerous-triggers — High
use of fundamentally insecure workflow trigger
File: .github/workflows/backport.yaml
Fix guidance: https://docs.zizmor.sh/audits/#dangerous-triggers
Locations:
- Line 2–4 (cols 0–32) — pull_request_target is almost always used insecurely
unpinned-uses — High
unpinned action reference
File: .github/workflows/backport.yaml
Fix guidance: https://docs.zizmor.sh/audits/#unpinned-uses
Locations:
- Line 19 (cols 14–53) — expression
sorenlouv/backport-github-action@v9.5.1— action is not pinned to a hash (required by blanket policy)
template-injection — High (13 similar finding(s))
code injection via template expansion
File: .github/workflows/base-images.yaml
Fix guidance: https://docs.zizmor.sh/audits/#template-injection
Locations:
- Line 92–142 (cols 8–13) — this step
- Line 99 (cols 31–64) — expression
github.event.inputs.base_os_image— may expand into attacker-controllable code - Line 94 (cols 8–11) — this run block
- Line 100 (cols 33–68) — expression
github.event.inputs.registry_prefix— may expand into attacker-controllable code - Line 101 (cols 22–46) — expression
github.event.inputs.arch— may expand into attacker-controllable code - Line 102 (cols 23–48) — expression
github.event.inputs.model— may expand into attacker-controllable code - Line 103 (cols 32–66) — expression
github.event.inputs.kairos_version— may expand into attacker-controllable code - Line 104 (cols 30–62) — expression
github.event.inputs.trusted_boot— may expand into attacker-controllable code - …and 9 more location(s) in this file.
unpinned-uses — High (8 similar finding(s))
unpinned action reference
File: .github/workflows/base-images.yaml
Fix guidance: https://docs.zizmor.sh/audits/#unpinned-uses
Locations:
- Line 153 (cols 14–33) — expression
actions/checkout@v4— action is not pinned to a hash (required by blanket policy) - Line 156 (cols 14–43) — expression
docker/setup-buildx-action@v3— action is not pinned to a hash (required by blanket policy) - Line 162 (cols 14–35) — expression
docker/bake-action@v6— action is not pinned to a hash (required by blanket policy) - Line 184 (cols 14–40) — expression
actions/upload-artifact@v4— action is not pinned to a hash (required by blanket policy) - Line 294 (cols 14–33) — expression
actions/checkout@v4— action is not pinned to a hash (required by blanket policy) - Line 297 (cols 14–43) — expression
docker/setup-buildx-action@v3— action is not pinned to a hash (required by blanket policy) - Line 353 (cols 14–40) — expression
actions/upload-artifact@v4— action is not pinned to a hash (required by blanket policy) - Line 365 (cols 14–42) — expression
actions/download-artifact@v4— action is not pinned to a hash (required by blanket policy)
Please review these findings before merging.
There was a problem hiding this comment.
Summary
| Severity | Count |
|---|---|
| High | 23 |
| Total | 23 |
Details
Grouped by audit rule and file. Line/column refer to the workflow or action YAML on the scanned branch.
dangerous-triggers — High
use of fundamentally insecure workflow trigger
File: .github/workflows/backport.yaml
Fix guidance: https://docs.zizmor.sh/audits/#dangerous-triggers
Locations:
- Line 2–4 (cols 0–32) — pull_request_target is almost always used insecurely
unpinned-uses — High
unpinned action reference
File: .github/workflows/backport.yaml
Fix guidance: https://docs.zizmor.sh/audits/#unpinned-uses
Locations:
- Line 19 (cols 14–53) — expression
sorenlouv/backport-github-action@v9.5.1— action is not pinned to a hash (required by blanket policy)
template-injection — High (13 similar finding(s))
code injection via template expansion
File: .github/workflows/base-images.yaml
Fix guidance: https://docs.zizmor.sh/audits/#template-injection
Locations:
- Line 92–142 (cols 8–13) — this step
- Line 99 (cols 31–64) — expression
github.event.inputs.base_os_image— may expand into attacker-controllable code - Line 94 (cols 8–11) — this run block
- Line 100 (cols 33–68) — expression
github.event.inputs.registry_prefix— may expand into attacker-controllable code - Line 101 (cols 22–46) — expression
github.event.inputs.arch— may expand into attacker-controllable code - Line 102 (cols 23–48) — expression
github.event.inputs.model— may expand into attacker-controllable code - Line 103 (cols 32–66) — expression
github.event.inputs.kairos_version— may expand into attacker-controllable code - Line 104 (cols 30–62) — expression
github.event.inputs.trusted_boot— may expand into attacker-controllable code - …and 9 more location(s) in this file.
unpinned-uses — High (8 similar finding(s))
unpinned action reference
File: .github/workflows/base-images.yaml
Fix guidance: https://docs.zizmor.sh/audits/#unpinned-uses
Locations:
- Line 153 (cols 14–33) — expression
actions/checkout@v4— action is not pinned to a hash (required by blanket policy) - Line 156 (cols 14–43) — expression
docker/setup-buildx-action@v3— action is not pinned to a hash (required by blanket policy) - Line 162 (cols 14–35) — expression
docker/bake-action@v6— action is not pinned to a hash (required by blanket policy) - Line 184 (cols 14–40) — expression
actions/upload-artifact@v4— action is not pinned to a hash (required by blanket policy) - Line 294 (cols 14–33) — expression
actions/checkout@v4— action is not pinned to a hash (required by blanket policy) - Line 297 (cols 14–43) — expression
docker/setup-buildx-action@v3— action is not pinned to a hash (required by blanket policy) - Line 353 (cols 14–40) — expression
actions/upload-artifact@v4— action is not pinned to a hash (required by blanket policy) - Line 365 (cols 14–42) — expression
actions/download-artifact@v4— action is not pinned to a hash (required by blanket policy)
Please review these findings before merging.
Main's .github/workflows/base-images.yaml is a stub (echo "Hello"). The real kairosify workflow lives on the long-lived origin/kairos-init branch; port the 3 files needed to dispatch base-image builds: .github/workflows/base-images.yaml (kairosify matrix, 188 lines) docker-bake-kairosify.hcl (bake target) dockerfiles/kairosify/Dockerfile.kairosify (kairos-init + BASE_OS_IMAGE) Defaults bumped to match PE-8787's target versions: kairos_init_image: v0.8.12 → v0.16.1 kairos_version: v4.0.1 → v4.1.2 registry_prefix: pe-8215/… → pe-8787/… docker-bake-kairosify.hcl KAIROS_INIT_IMAGE v0.8.12 → v0.16.1 docker-bake-kairosify.hcl KAIROS_VERSION v3.5.9 → v4.1.2 Covers kairosify targets only (Ubuntu 22/24, OpenSUSE 15.6, SLEM 5.4/5.5, Ubuntu 24.04 UKI). RHEL / RHEL FIPS / Ubuntu FIPS are handled by PR #681 (feat/unified-base-images) — kept separate here to preserve review scope.
Summary
Unified base-image workflow that produces every OS variant a release needs in a single
workflow_dispatch, with credentials handled via age-encrypted inputs so they never appear in cleartext anywhere.Full-house dispatch (
fips=true+trusted_boot=true+ all creds) produces 12 kairosified base images pushed tous-east1-docker.pkg.dev/…/edge/kairos-*:Empirically verified end-to-end in run 29276504878 — 12/12 images built and pushed, credential audit clean.
What this PR does
1. Extends the existing kairos-init
base-images.yamlgenerate-matrix+kairosifyjobs are preserved unchanged — samedocker/bake-action+docker-bake-kairosify.hclpipeline that already produced Ubuntu/OpenSUSE/SLEM/UKI images successfully on this branch.Adds two new jobs:
generate-extensions-matrix— computes RHEL + Ubuntu FIPS matrix rows based on which credentials the dispatcher supplied and whetherfips=true.extensions— matrix-parallel builder that runsdocker buildper RHEL/Ubuntu-FIPS variant using BuildKit--secretmounts.collect-outputsnow aggregates tags from both matrices.Also:
trusted_boot=trueis now additive (adds Ubuntu 24 UKI on top of the standard 5) rather than replacing them.fipscheckbox mirrors thetrusted_bootcheckbox — check it to expand the matrix with FIPS variants.2. Encrypt-at-dispatch credential handling
Instead of raw
rhel_subscription_password, users paste age-encrypted ciphertexts. Ciphertexts are useless without the private key, which lives only as theWORKFLOW_DECRYPT_KEYrepo secret and is auto-masked by GitHub.Plaintext credentials never leave the operator's laptop.
Test plan
yaml.safe_loadparsesbase-images.yamlDECRYPT_KEYpath → hard error with actionable message