Vipsharm gpu#676
Open
Medhakulam wants to merge 23 commits into
Open
Conversation
…ped GPU Operator Add scripts/install-nvidia-drivers.sh to bake the NVIDIA data-center driver and its DKMS kernel modules into the Ubuntu base image, so GPU nodes can run the NVIDIA GPU Operator with driver.enabled=false in fully air-gapped environments (no host-side network, no operator driver container). The script derives the target kernel from /lib/modules (not the builder's uname -r), installs ABI-exact headers via install-kernel-headers.sh, forces DKMS build/install/depmod against that kernel, and verifies nvidia*.ko landed in the image kernel tree. It also blacklists nouveau, autoloads the nvidia stack, enables nvidia-persistenced, and rebuilds the initrd. Wire it into the base-image target (Ubuntu/amd64, after the kernel is finalized) behind INSTALL_NVIDIA_GPU_DRIVERS, with tunable branch/type/repo/fabricmanager/ toolkit args. Document in .arg.template and docs/nvidia-gpu-airgapped.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…section Explain that linux-headers/linux-image/linux-modules listed by NVIDIA belong to the driver-container + local-apt-mirror strategy (driver.enabled=true), whereas this integration uses the pre-installed-driver model (driver.enabled=false, no mirror) and satisfies those three requirements at build time in the image. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… semantics Clarify that the installer is version-agnostic (derives CUDA repo tag and kernel codename from the image's os-release), so it works on both Ubuntu 22.04 and 24.04. Note that NVIDIA_DRIVER_BRANCH selects a driver branch (latest patch within it), not an exact point release. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… Operator Add scripts/install-amdgpu-drivers.sh, the AMD counterpart to the NVIDIA installer. Bakes the amdgpu-dkms kernel driver into the Ubuntu base image so AMD Instinct nodes can run the AMD GPU Operator with driver.enable=false in fully air-gapped environments (no host-side network, no operator-built driver). Reuses the same build-time approach: derive the image kernel from /lib/modules (not the builder's uname -r), install ABI-exact headers via install-kernel-headers.sh plus linux-modules-extra, register the AMD driver repo via the version-matched amdgpu-install package (auto-discovered from repo.radeon.com/amdgpu-install/<version>/), install amdgpu-dkms, force DKMS build/install/depmod against the image kernel, and verify amdgpu.ko landed. No nouveau-style blacklist is needed since amdgpu-dkms replaces the in-tree amdgpu module. Wire it into the base-image target behind INSTALL_AMD_GPU_DRIVERS with an AMDGPU_ROCM_VERSION arg (default 7.2.4), alongside a build-time guard making the NVIDIA and AMD pre-install paths mutually exclusive. Document in .arg.template, docs/amd-gpu-airgapped.md, and cross-link from the NVIDIA doc. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e-installed Add a "Building the air-gapped content" section to both the NVIDIA and AMD guides. Clarify that pre-installing the driver only lets you drop the driver image (NVIDIA) or the KMM/kaniko/ubuntu build images (AMD) -- the rest of the operator still ships as container images that must be mirrored into the content bundle. Cover the easy-to-miss pieces: NVIDIA still needs container-toolkit (unless also pre-installed on host) and the CUDA validation image, plus the k3s/rke2 containerd config env overrides for Palette Edge; AMD needs cert-manager installed first, a DeviceConfig CR with driver.enable=false, and node labelling. Recommend enumerating exact image tags via 'helm template' and note Palette content-bundle additional-images handling. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bump the default NVIDIA driver branch from 570 to 580 across the Earthfile ARG, install-nvidia-drivers.sh, .arg.template, and docs. Refresh point-release examples to 580.159.03. The 570 entries left in "e.g. 550, 570, 580" lists are valid-branch examples, not defaults. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… hang On modern NVIDIA data-center GPUs (Ada/Hopper/Blackwell — reproduced on HPE ProLiant Compute DL380a Gen12 with a GH100-class GPU), a CanvOS image built with INSTALL_NVIDIA_GPU_DRIVERS=true would hang forever at first boot inside systemd-udev-settle. Console showed a kernel stack trace ending at gh100_gsp_oneinit+0x2cf/0x440 [nouveau]: initramfs udev auto-loaded nouveau before the rootfs modprobe.d blacklist could apply, and nouveau's experimental GSP-RM path hung on the GPU. udev-settle never returned; the node never came up; nvidia.ko never got to load. Three changes, together sufficient to unbrick boot: * Earthfile: when INSTALL_NVIDIA_GPU_DRIVERS is true, append rd.driver.blacklist=nouveau modprobe.blacklist=nouveau nouveau.modeset=0 to /etc/cos/bootargs.cfg. This is the actual load-bearing fix: rd.driver.blacklist= is honored by dracut/initramfs *before* any udev event fires, so nouveau never binds. Mirrors the existing cgroup-v2 append pattern immediately above. * install-nvidia-drivers.sh: write /etc/dracut.conf.d/95-blacklist-nouveau.conf so BOTH the script's own dracut rebuild AND the later Earthfile-driven dracut rebuild (a) omit nouveau.ko from the initrd entirely and (b) carry the modprobe blacklist file into initramfs. Belt-and-braces in case the cmdline args are ever stripped downstream. * install-nvidia-drivers.sh: drop nvidia_drm from /etc/modules-load.d/nvidia.conf. On headless GPU nodes it serves no purpose and loading it at early boot has a class of KMS/DRM-conflict failure modes. Loads on demand if anything actually needs it. Follow-up: the UKI build path (--extend-cmdline in the build-uki targets) also needs the same nouveau cmdline args when INSTALL_NVIDIA_GPU_DRIVERS is true. Not addressed here since field failures were non-UKI grub boot; worth adding in a subsequent commit. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…upport
Flip the default flavor of the pre-installed NVIDIA kernel modules from
"proprietary" to "open". The closed modules fail RmInitAdapter (0x22:0x56:897)
on Hopper (H100/H200) and Blackwell (RTX PRO 6000 Blackwell, B100/B200/GB200);
`nvidia-smi` on such a host reports "No devices were found" and the entire
GPU Operator stack downstream stays wedged (toolkit init loops, containerd
never registers the nvidia runtime, device plugin never advertises
nvidia.com/gpu, workload pods stay Pending on Insufficient nvidia.com/gpu).
The open modules also work fine on Turing, Ampere, and Ada, so the new
default is safe for every server-class NVIDIA GPU generation Turing and
newer -- essentially every card CanvOS is likely to target. Only Pascal /
Volta (V100/P100/P40) require the closed modules and now need an explicit
NVIDIA_DRIVER_TYPE=proprietary override.
Applied consistently across:
* scripts/install-nvidia-drivers.sh -- runtime default + header comment
* Earthfile -- ARG default
* .arg.template -- user-facing hint
* docs/nvidia-gpu-airgapped.md -- config table plus new "Choosing the
module flavor" section with a hardware/flavor matrix and the RmInitAdapter
symptom description for people googling the error.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The pre-install script silently shipped broken images when AMD's amdgpu-dkms
source did not build against the image kernel (observed on Ubuntu 24.04 HWE
6.17 with the previous 7.2.4 / 30.30.4 default, whose 6.16.13 source fails at
DKMS configure). The verify step accepted the in-tree amdgpu.ko.zst as
sufficient because it only searched anywhere under /lib/modules/$KVER/, and
apt-get failures were papered over. Nodes booted using the in-tree driver
instead of the DKMS one without any signal that this had happened.
Changes:
- Introduce AMDGPU_DRIVER_SOURCE={dkms,inbox}. dkms (default) DKMS-builds AMD's
out-of-tree amdgpu; inbox skips the AMD apt repo entirely and only ensures
the kernel's in-tree amdgpu autoloads. Any other value is a hard error.
- Rename AMDGPU_ROCM_VERSION -> AMDGPU_DRIVER_RELEASE (the URL segment is a
driver-release marker, not a ROCm x.y.z). Bump default 7.2.4 -> 31.30
(amdgpu-dkms 6.19.4, covers 22.04/24.04 kernels through 6.17).
- Fail hard in dkms mode: set -eo pipefail; remove '|| true' shims around
dkms build/install; drop 'dpkg --configure -a --force-all' silent-fallback;
verify amdgpu landed under $MODDIR/updates/dkms AND that dkms status shows
': installed' for the target kernel. Failure prints make.log tail and
points at bumping AMDGPU_DRIVER_RELEASE or switching to inbox.
- Write /etc/canvos/amdgpu-driver-source on-node marker so ops can query
which mode ran and, in dkms mode, the release + module/version + kernel.
- Docs: two-modes explainer up front, snapshot mapping table (2026-07-11)
linking to AMD's canonical system-requirements page, "when the DKMS build
fails" section documenting the bump-or-fallback flow.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…build container)
amdgpu-dkms >= 6.18 (observed on release 31.30 = amdgpu-dkms 6.19.4) invokes
mokutil from the DKMS sign_tool hook to enroll a Machine Owner Key. mokutil
reads /sys/firmware/efi/efivars, which is not exposed inside Docker/Earthly
RUN containers, so it aborts with:
"EFI variables are not supported on this system /
/sys/firmware/efi/efivars not found, aborting."
and the amdgpu-dkms postinst returns non-zero, failing the base-image build.
Drop /etc/dkms/framework.conf.d/canvos-no-mok-signing.conf with sign_tool=""
immediately before apt-get install amdgpu-dkms. DKMS honors this and skips
signing entirely, sidestepping the mokutil invocation. Modules are unsigned;
this path already documents that Secure Boot / UKI is unsupported.
The step-5 error message now lists this as the third common failure mode so
future regressions surface with a pointer at the drop-in.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…tor v1.5.0) Empirical reproduction on the kairos-ubuntu:24.04-core-amd64-generic-v4.0.4 base image (kernel 6.17.0-35-generic) with the AMDGPU_ROCM_VERSION=7.2.4-era script + Abhijit's masking reverted showed that amdgpu-dkms 6.16.13 builds cleanly against Linux 6.17: DKMS reports the module `installed`, all 8 amd*.ko files land under /lib/modules/<kver>/updates/dkms/, and apt-get returns 0. The "EFI variables are not supported on this system / /sys/firmware/efi/efivars not found, aborting" line printed during the 30.30.x postinst is a cosmetic mokutil warning (sign_tool step); the postinst continues past it and finishes normally. The signing config drop-in from ad4c6fa is still valuable as defensive belt-and-suspenders for the 31.x line where the same message is treated as fatal. Given that, revert the previous 31.30 default (which crossed AMD's tech- preview boundary) back to 7.2.1 -- the release AMD's GPU Operator v1.5.0 release notes explicitly pair the operator with. AMD publishes both ROCm-alias and driver-release-marker URL segments at repo.radeon.com/amdgpu-install/, so "7.2.1" is a valid AMDGPU_DRIVER_RELEASE (equivalent to "30.30.1"). Also refactor the docs' "Version alignment" section to lead with the operator → ROCm → driver → kernel chain and explicitly label the 31.x line as tech-preview. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…lure Earthly's -i interactive debugger frequently hands out a broken tty in the buildkit sandbox, making /var/lib/dkms/*/build/make.log unreachable when amdgpu-dkms's postinst fails. Instead of asking users to debug interactively, tail the make.log(s) and a few environment probes (kernel headers state, Module.symvers presence, sign_tool drop-in state, MemAvailable) to stderr before die-ing. The next non-interactive rebuild surfaces the underlying compile error directly in the Earthly output. No behavior change on the happy path. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
amdgpu-dkms's ./configure runs a Kbuild probe to detect CFLAGS. In buildkit's
default unprivileged sandbox that probe fails with
configure: error: cannot detect CFLAGS...
grep: amd/dkms/config/config.h: No such file or directory
make: *** No rule to make target 'amd/dkms/config/config.h'
Reproduced against a plain `docker run --privileged` baseline: same script,
same base image (kairos-ubuntu:24.04-core-amd64-generic-v4.0.4), same kernel
(6.17.0-35-generic), same AMDGPU_DRIVER_RELEASE=7.2.1 -- succeeds end-to-end
with modules landing under /lib/modules/<kver>/updates/dkms/. The only
observed difference is --privileged, so mark the AMD RUN step explicitly.
The wrapper (./earthly.sh) already passes --allow-privileged, and other RUN
steps in this repo use --privileged similarly, so no downstream change needed.
NVIDIA's driver install doesn't use the same ./configure sanity-check step
and continues to work unprivileged; its RUN block is left as-is.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
amdgpu-dkms's ./configure fails silently at "cannot detect CFLAGS..." when Kbuild can't complete the `make -f -` heredoc probe it uses to extract KBUILD_CFLAGS. build-essential covers gcc/make/libc but modern kernel Makefiles unconditionally reach for bc, bison, flex, libelf-dev, libssl-dev, and pahole (from dwarves). --no-install-recommends skips all of those. Enumerate them explicitly to keep the image lean without breaking the AMD DKMS build. dwarves is AMD's own `Recommends:` on amdgpu-dkms -- it just wasn't pulled because we use --no-install-recommends. cpio + xz-utils are added defensively for the initramfs trigger that runs after DKMS install. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ild) Compiles amdgpu-dkms against the kairos base image's kernel in a host-side `docker run --privileged` container, then tars the produced kernel module + firmware + config drop-ins into build/amdgpu-artifact-<release>-<kver>-<digest>.tar.gz. The main Earthly build consumes that artifact via COPY, sidestepping the buildkit RUN sandbox limitation that breaks AMD's ./configure heredoc probe. Cache by (base-image digest x kernel version x AMDGPU_DRIVER_RELEASE); repeat builds skip the ~8-10 min compile. Cache miss reproduces the same behavior we already verified end-to-end via direct docker run on the jump host. Prints "AMDGPU_ARTIFACT_PATH=<abs-path>" on the last stdout line so the earthly.sh wrapper (next commit) can pick it up and thread it into Earthly. Standalone; no other files touched by this commit. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Before dispatching to earthly (build_with_proxy/build_without_proxy), detect INSTALL_AMD_GPU_DRIVERS=true + AMDGPU_DRIVER_SOURCE=dkms from either .arg or CLI overrides, derive BASE_IMAGE via the same tag formula the Earthfile uses, and invoke scripts/prebuild-amdgpu-artifact.sh to compile the amdgpu module on the host. The last line of prebuild output is parsed for the artifact path; it's threaded to earthly as --AMDGPU_ARTIFACT_PATH=<abs>, which the companion Earthfile change consumes via COPY + tar-extract. Inbox mode skips this step entirely. Non-AMD builds skip this step entirely. Existing CLI/.arg semantics for INSTALL_AMD_GPU_DRIVERS, AMDGPU_DRIVER_SOURCE, AMDGPU_DRIVER_RELEASE, and BASE_IMAGE are preserved unchanged. Ubuntu-only guard: prebuild refuses OS_DISTRIBUTION != ubuntu with a clear error, mirroring the mutual-exclusion / Ubuntu-only checks already in the Earthfile for the AMD block. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…+ docs
Complete the prebuild-artifact flow started in the previous two commits.
Earthfile:
* new ARG AMDGPU_ARTIFACT_PATH (empty default)
* base-image AMD block now branches on (AMDGPU_DRIVER_SOURCE, artifact):
- dkms + artifact present -> COPY tarball, extract, depmod, initrd
(fast path used by ./earthly.sh + prebuild helper -- the release path)
- inbox, or dkms without artifact -> unchanged legacy path (works under
`docker run --privileged`; fails under buildkit's RUN sandbox at
AMD's ./configure -- documented)
scripts/install-amdgpu-drivers.sh:
* new AMDGPU_ARTIFACT_PATH fast path before the in-buildkit DKMS block:
extract tarball, verify updates/dkms/amdgpu.ko*, depmod, rebuild initrd,
write the on-node marker at /etc/canvos/amdgpu-driver-source. In-buildkit
DKMS legacy path retained for direct `docker run` invocations and for
scripts/prebuild-amdgpu-artifact.sh's own inner container.
* docstring updated to describe the three execution paths.
docs/amd-gpu-airgapped.md:
* new "two-stage build" explanation covering why the prebuild exists
(buildkit RUN sandbox breaks AMD ./configure) and how it's wired up
behind ./earthly.sh.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…d'd) The prebuild's inner container bind-mounts the install scripts read-only (-v ...:ro). chmod +x fails on RO mounts. Bash can execute scripts without +x when invoked as `bash <path>`; no functional difference otherwise. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Previous verify path deleted the tarball via STAGE_DIR trap before the user
could inspect it. Now:
* mv to final path FIRST (before verify), so any failure leaves the file
behind for inspection
* on verify failure, dump the first 40 tar entries to stderr so we can
see what path structure actually landed in the archive
* grep pattern loosened to `updates/dkms/.*amdgpu\.ko` (regex) in case
tar mangled paths in an unexpected way
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ld context) Earthly's COPY expects paths relative to the Earthfile's directory (the repo root), not absolute host paths. The prebuild helper emits an absolute path for its own file-existence checks; the wrapper now converts that to a repo-relative path before threading it into Earthly as --AMDGPU_ARTIFACT_PATH=build/amdgpu-artifact-...tar.gz. Fails fast with a clear message if the artifact somehow lands outside the repo tree. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
`build/*` in .earthlyignore was hiding the pre-built AMD driver tarball from Earthly's build context, so COPY build/amdgpu-artifact-...tar.gz failed with "no such file or directory" even though the artifact existed on the host. Add a negation pattern for build/amdgpu-artifact-*.tar.gz so this specific artifact is visible while everything else under build/ (Earthly SAVE ARTIFACT outputs, etc.) stays ignored. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ueue
Reported: node with baked-in DKMS amdgpu (8x MI325X) fails to boot -- drops
into dracut-emergency after `dracut-initqueue: Timed out for waiting the udev
queue being empty`. Screenshots show amdgpu still emitting per-XCP-partition
and per-ring init messages long after boot has given up.
Root cause: rebuilding the initrd (default true) pulled the DKMS amdgpu
modules under /lib/modules/<kver>/updates/dkms/ into the initrd. amdgpu then
loads *before* switch-root, and the sheer volume of udev events it emits on
multi-GPU MI silicon (8 GPUs x 8 XCP partitions + JPEG rings + SMU/PSP/KFD
handshakes) keeps systemd-udev-settle from ever seeing an empty queue.
initqueue times out (~5 min), boot fails, node reboots, loops.
amdgpu is not required to mount rootfs -- NVMe/SATA use their own drivers.
Load it *after* switch-root via /etc/modules-load.d/amdgpu.conf (which the
script was already writing), where there is no timeout pressure. This
matches kairos-a176's observed behavior with the in-tree driver.
Two guards:
1. Flip default AMDGPU_REBUILD_INITRD to `false` (was `true`) in
Earthfile, .arg.template, and install script.
2. Even when someone sets AMDGPU_REBUILD_INITRD=true, drop
/etc/dracut.conf.d/98-canvos-amdgpu-omit.conf with
`omit_drivers+=" amdgpu amdttm amdkcl amd-sched amddrm_ttm_helper
amddrm_buddy amddrm_exec amdxcp "` so dracut still leaves amdgpu out.
Applied consistently across all three code paths in install-amdgpu-drivers.sh
(artifact fast path, inbox mode, in-buildkit DKMS fallback).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.