[TRTLLMINF-213][infra] Artifactory container image migration by tburt-nv · Pull Request #16970 · NVIDIA/TensorRT-LLM

tburt-nv · 2026-07-28T19:42:17Z

Dev Engineer Review

Migrated Docker image references and publishing from urm.nvidia.com to artifactory.nvidia.com.
Added Artifactory image-pull secrets, Docker credentials, and PyPI mirror support.
Updated Enroot image normalization and Docker documentation.
Added SLURM resource registration, orphan cleanup, frontend failover, and retry credential handling.
Review focus: validate credential IDs, secret scope, image tags, registry paths, cleanup behavior, and SLURM retry handling.
CI runs reported failures. The provided information does not identify the failure causes.

QA Engineer Review

No test changes.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

tburt-nv · 2026-07-28T19:42:33Z

/bot run --extra-stage "Build-Docker-Images"

tensorrt-cicd · 2026-07-28T19:49:00Z

PR_Github #62283 [ run ] triggered by Bot. Commit: be9843d Link to invocation

tensorrt-cicd · 2026-07-28T20:00:08Z

PR_Github #62283 [ run ] completed with state FAILURE. Commit: be9843d
/LLM/main/L0_MergeRequest_PR pipeline #50450 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

tburt-nv · 2026-07-29T00:40:28Z

/bot run

tensorrt-cicd · 2026-07-29T00:46:01Z

PR_Github #62349 [ run ] triggered by Bot. Commit: be9843d Link to invocation

tensorrt-cicd · 2026-07-29T01:47:26Z

PR_Github #62349 [ run ] completed with state FAILURE. Commit: be9843d
/LLM/main/L0_MergeRequest_PR pipeline #50516 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

tburt-nv · 2026-07-29T20:58:14Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-29T21:05:05Z

PR_Github #62603 [ run ] triggered by Bot. Commit: 62dfabd Link to invocation

tensorrt-cicd · 2026-07-29T22:36:29Z

PR_Github #62603 [ run ] completed with state SUCCESS. Commit: 62dfabd
/LLM/main/L0_MergeRequest_PR pipeline #50744 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

tburt-nv · 2026-07-30T16:32:31Z

/bot run --disable-fail-fast

tburt-nv · 2026-07-30T17:54:53Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-30T18:01:30Z

PR_Github #62823 [ run ] triggered by Bot. Commit: 6f0f180 Link to invocation

tensorrt-cicd · 2026-07-30T18:43:43Z

PR_Github #62823 [ run ] completed with state ABORTED. Commit: 6f0f180

Link to invocation

tburt-nv · 2026-07-30T18:59:03Z

/bot run --extra-stage "Build-Docker-Images"

tburt-nv · 2026-07-30T18:59:37Z

/bot run --extra-stage "Build-Docker-Images" --disable-fail-fast

tensorrt-cicd · 2026-07-30T19:06:31Z

PR_Github #62833 [ run ] triggered by Bot. Commit: c450800 Link to invocation

tensorrt-cicd · 2026-07-30T21:43:35Z

PR_Github #62833 [ run ] completed with state FAILURE. Commit: c450800
/LLM/main/L0_MergeRequest_PR pipeline #50957 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

tburt-nv · 2026-07-30T22:25:25Z

/bot run --extra-stage "Build-Docker-Images" --disable-fail-fast

tensorrt-cicd · 2026-07-30T22:31:59Z

PR_Github #62864 [ run ] triggered by Bot. Commit: 9053ff3 Link to invocation

tensorrt-cicd · 2026-07-31T04:28:42Z

PR_Github #62864 [ run ] completed with state SUCCESS. Commit: 9053ff3
/LLM/main/L0_MergeRequest_PR pipeline #50986 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

tburt-nv · 2026-07-31T17:07:50Z

/bot run

tensorrt-cicd · 2026-07-31T17:15:54Z

PR_Github #63076 [ run ] triggered by Bot. Commit: 112382a Link to invocation

tensorrt-cicd · 2026-07-31T19:47:10Z

PR_Github #63076 [ run ] completed with state FAILURE. Commit: 112382a
/LLM/main/L0_MergeRequest_PR pipeline #51169 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

tburt-nv · 2026-08-03T15:49:57Z

/bot run --disable-fail-fast

coderabbitai · 2026-08-05T18:49:28Z

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

tburt-nv · 2026-08-05T18:50:44Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-08-05T18:57:20Z

PR_Github #64100 [ run ] triggered by Bot. Commit: 8a89113 Link to invocation

tensorrt-cicd · 2026-08-05T21:29:13Z

PR_Github #64100 [ run ] completed with state SUCCESS. Commit: 8a89113
/LLM/main/L0_MergeRequest_PR pipeline #52024 completed with status: 'FAILURE'

CI Report

⚠️ Multi-GPU Label Required:
Multi-GPU tests require the ci: full pre-merge approved label on this PR. Ask a member of NVIDIA/trt-llm-ci-approvers to add the label, then re-trigger CI with the same bot command (no rebase needed).

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Signed-off-by: Tyler Burt <195370667+tburt-nv@users.noreply.github.com>

coderabbitai · 2026-08-05T23:20:15Z

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

tburt-nv · 2026-08-05T23:20:18Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-08-05T23:26:34Z

PR_Github #64137 [ run ] triggered by Bot. Commit: 9180ca4 Link to invocation

tensorrt-cicd · 2026-08-06T05:24:59Z

PR_Github #64137 [ run ] completed with state FAILURE. Commit: 9180ca4
/LLM/main/L0_MergeRequest_PR pipeline #52057 completed with status: 'FAILURE'

CI Report

⚠️ Multi-GPU Label Required:
Multi-GPU tests require the ci: full pre-merge approved label on this PR. Ask a member of NVIDIA/trt-llm-ci-approvers to add the label, then re-trigger CI with the same bot command (no rebase needed).

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

tburt-nv · 2026-08-06T14:30:57Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-08-06T14:37:29Z

PR_Github #64359 [ run ] triggered by Bot. Commit: 9180ca4 Link to invocation

tensorrt-cicd · 2026-08-06T15:59:19Z

PR_Github #64359 [ run ] completed with state SUCCESS. Commit: 9180ca4
/LLM/main/L0_MergeRequest_PR pipeline #52253 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

tburt-nv · 2026-08-06T16:00:40Z

/bot run --disable-fail-fast

brnguyen2

Migration itself looks mechanically complete for the Jenkins/K8s side (image refs, imagePullSecrets, credential IDs). Two things worth settling before merge, plus a couple of leftovers.

Enroot URI conversion is now host-specific — is that deliberate? [L0_Test.groovy:1806](https://github.com/NVIDIA/TensorRT-LLM/pull/16970/files#diff-3efa8a6083a90a6f716e4d4bc81710c8104f8bfc5ae642e9fef44df8061e6b7dR1806) and :1011 replaced the unconditional urm.nvidia.com/ → urm.nvidia.com# rewrite with an artifactory-only one. env.dockerImage is a job parameter, so a run pointed at a urm.nvidia.com/... image now gets docker://urm.nvidia.com/sw-tensorrt-docker/..., which enroot parses as a Docker Hub path (no # → default registry).

If the intent is that urm.nvidia.com is being retired under TRTLLMINF-213 and any urm-targeted run is dead regardless, that's a reasonable call — just say so in a comment and I'll drop this. Two things make me unsure that's the case: the DLFW_IMAGE constant near the top of this file still points at urm.nvidia.com/docker/nvidia/pytorch, and run_disagg.sh deliberately kept both rewrites in this same PR. If urm stays pullable through a transition period, the narrowed rewrite turns a working override into a confusing enroot parse failure (or a Docker Hub lookup) instead of a clean auth error. A generic first-segment host/ → host# costs nothing and is retirement-agnostic either way.

Undisclosed leftovers. The PR description doesn't mention the PIP_INDEX_URL behavior change in Dockerfile.multi beyond "PyPI mirror support" — see inline. Also still on urm.nvidia.com/sw-tensorrt-docker: examples/layer_wise_benchmarks/slurm_init_containers.sh:29 and jenkins/scripts/perf/local/configs/example.conf:38, both user-facing examples that break once images stop being published to the old repo. If the old repo stays readable for a while, fine to defer — but say so in the description. (Same question underneath: what's the actual timeline for urm going away?)

Docs. docker/README.md is updated; nothing else in docs/ seems to name the registry, so I think that's covered.

Ticket tag (TRTLLMINF-213) is correct for this. The slurm_job_result.txt delimiter change (' ' → '|') is a good fix but is unrelated to the registry migration and isn't mentioned in the description — worth one line there so a future bisect knows why it moved.

Non-blocking overall; the enroot URI question is the one I'd want answered before merge.

brnguyen2 · 2026-08-06T16:21:29Z

+                def container = LLM_DOCKER_IMAGE
+                if (cluster.containerRuntime.toString() == "ENROOT") {
+                    container = LLM_DOCKER_IMAGE
+                        .replace("${ARTIFACTORY_DOCKER_HOST}/", "${ARTIFACTORY_DOCKER_HOST}#")


This narrows the previous unconditional urm.nvidia.com/ → urm.nvidia.com# rewrite to the artifactory host only. LLM_DOCKER_IMAGE comes from env.dockerImage, which is an overridable job parameter — a run pointed at a urm.nvidia.com/... image now yields docker://urm.nvidia.com/sw-tensorrt-docker/tensorrt-llm:tag. Enroot's URI grammar is docker://[USER@][REGISTRY#]IMAGE[:TAG], so without the # the whole thing is treated as an image path on the default registry.

Is that intentional — i.e. urm is being retired as part of this migration, so a urm-targeted run would fail on auth/availability anyway and there's no point preserving the rewrite? If so, a one-line comment here saying that would save the next reader the same question. What makes me unsure: DLFW_IMAGE near the top of this file still resolves against urm.nvidia.com/docker/..., and jenkins/scripts/perf/local/run_disagg.sh kept both rewrites in this PR — so at least some urm paths are expected to keep working.

If urm stays pullable during a transition, a generic first-segment host/ → host# rewrite handles both without a host list and survives the next registry move too. Same at [L0_Test.groovy:1014](https://github.com/NVIDIA/TensorRT-LLM/pull/16970/files#diff-3efa8a6083a90a6f716e4d4bc81710c8104f8bfc5ae642e9fef44df8061e6b7dR1014).

urm.nvidia.com will not be used for pulling images after this PR is merged.

brnguyen2 · 2026-08-06T16:21:29Z

 ARG BUILD_WHEEL_ARGS="--clean"
 ARG BUILD_WHEEL_SCRIPT="scripts/build_wheel.py"
 RUN --mount=type=cache,target=/root/.cache/pip --mount=type=cache,target=${CCACHE_DIR} \
+    if [ -n "$GITHUB_MIRROR" ]; then \


Using GITHUB_MIRROR as the toggle for PIP_INDEX_URL overloads a flag whose documented meaning is "proxy for GitHub downloads". An external user who sets GITHUB_MIRROR to their own GitHub proxy silently gets their pip index redirected to an NVIDIA-internal host they can't reach, and the build fails at wheel install with a confusing error. It also hardcodes urm.nvidia.com in the PR that migrates off urm.nvidia.com — so this will need touching again (unless the pypi-remote side of urm is staying put; if so, worth a comment saying the docker repo and the pypi mirror have different fates).

Suggest a dedicated ARG PIP_INDEX_URL="" plumbed from docker/Makefile (mirroring the existing $(if $(GITHUB_MIRROR), --build-arg ...) pattern at docker/Makefile:104), with [jenkins/BuildDockerImage.groovy:302](https://github.com/NVIDIA/TensorRT-LLM/pull/16970/files#diff-3b1526ac2c26deffd5ac687c4acdc418ce272d866ad0867755e39072ebc8fef9R302) passing both. Same comment for the duplicate block at line 131.

It's already overloaded, and I don't intend to refactor this now.

brnguyen2 · 2026-08-06T16:21:29Z

-                trtllm_utils.llmExecStepWithRetry(this, script: "docker login urm.nvidia.com -u ${USERNAME} -p ${PASSWORD}")
+            // Read-write artifactory credentials (image push)
+            withCredentials([usernamePassword(credentialsId: "aws-artifactory-credentials", usernameVariable: 'USERNAME', passwordVariable: 'PASSWORD')]) {
+                trtllm_utils.llmExecStepWithRetry(this, script: "docker login artifactory.nvidia.com -u ${USERNAME} -p ${PASSWORD}")


The pull-side login in L0_Test.groovy moved to --password-stdin, but this push-side one still passes -p ${PASSWORD} on the command line — docker itself warns about this on every invocation, and it's a line this PR is already touching. Same one-liner applies:

trtllm_utils.llmExecStepWithRetry(this, script: "set +x; echo \"\$PASSWORD\" | docker login artifactory.nvidia.com -u \"\$USERNAME\" --password-stdin")

This is pre-existing behavior I'm not interested in refactoring as part of this PR. We can handle this enhancement separately.

tburt-nv · 2026-08-07T15:52:05Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-08-07T15:58:16Z

PR_Github #64636 [ run ] triggered by Bot. Commit: 9180ca4 Link to invocation

tensorrt-cicd · 2026-08-07T17:56:21Z

PR_Github #64636 [ run ] completed with state FAILURE. Commit: 9180ca4
/LLM/main/L0_MergeRequest_PR pipeline #52496 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

github-actions Bot assigned tburt-nv Jul 28, 2026

tburt-nv force-pushed the user/tburt/rt_image_migration branch from 62dfabd to 3292911 Compare July 30, 2026 16:32

tburt-nv force-pushed the user/tburt/rt_image_migration branch from 3292911 to 6f0f180 Compare July 30, 2026 17:54

tburt-nv force-pushed the user/tburt/rt_image_migration branch from 6f0f180 to c450800 Compare July 30, 2026 18:58

tburt-nv mentioned this pull request Jul 31, 2026

[None][infra] Always install latest openssl, libssl3t64 #17096

Merged

1 task

tburt-nv force-pushed the user/tburt/rt_image_migration branch from 9053ff3 to 112382a Compare July 31, 2026 17:07

tburt-nv added 10 commits August 5, 2026 16:19

use pypi mirror in wheel build

4dd3c42

Signed-off-by: Tyler Burt <195370667+tburt-nv@users.noreply.github.com>

fix cancelled job status reporting

78e7be2

Signed-off-by: Tyler Burt <195370667+tburt-nv@users.noreply.github.com>

migrate uploads

82b9b4e

Signed-off-by: Tyler Burt <195370667+tburt-nv@users.noreply.github.com>

update images

4b9c7f5

Signed-off-by: Tyler Burt <195370667+tburt-nv@users.noreply.github.com>

add creds for downloads

1df56e1

Signed-off-by: Tyler Burt <195370667+tburt-nv@users.noreply.github.com>

replace printf+ssh with scp

347f7bd

Signed-off-by: Tyler Burt <195370667+tburt-nv@users.noreply.github.com>

use stdin for docker login

6ee019a

Signed-off-by: Tyler Burt <195370667+tburt-nv@users.noreply.github.com>

propagate ENROOT_CONFIG_PATH through sbatch

c6c697c

Signed-off-by: Tyler Burt <195370667+tburt-nv@users.noreply.github.com>

comment credential IDs

4992de2

Signed-off-by: Tyler Burt <195370667+tburt-nv@users.noreply.github.com>

replace writeFile with heredoc

9180ca4

Signed-off-by: Tyler Burt <195370667+tburt-nv@users.noreply.github.com>

tburt-nv force-pushed the user/tburt/rt_image_migration branch from 8a89113 to 9180ca4 Compare August 5, 2026 23:20

tburt-nv requested a review from brnguyen2 August 6, 2026 16:09

brnguyen2 reviewed Aug 6, 2026

View reviewed changes

brnguyen2 added the ci: full pre-merge approved label Aug 7, 2026

dpitman-nvda approved these changes Aug 7, 2026

View reviewed changes

Uh oh!

Conversation

tburt-nv commented Jul 28, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dev Engineer Review

QA Engineer Review

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

tburt-nv commented Jul 28, 2026

Uh oh!

tensorrt-cicd commented Jul 28, 2026

Uh oh!

tensorrt-cicd commented Jul 28, 2026

Uh oh!

tburt-nv commented Jul 29, 2026

Uh oh!

tensorrt-cicd commented Jul 29, 2026

Uh oh!

tensorrt-cicd commented Jul 29, 2026

Uh oh!

tburt-nv commented Jul 29, 2026

Uh oh!

tensorrt-cicd commented Jul 29, 2026

Uh oh!

tensorrt-cicd commented Jul 29, 2026

Uh oh!

tburt-nv commented Jul 30, 2026

Uh oh!

tburt-nv commented Jul 30, 2026

Uh oh!

tensorrt-cicd commented Jul 30, 2026

Uh oh!

tensorrt-cicd commented Jul 30, 2026

Uh oh!

tburt-nv commented Jul 30, 2026

Uh oh!

tburt-nv commented Jul 30, 2026

Uh oh!

tensorrt-cicd commented Jul 30, 2026

Uh oh!

tensorrt-cicd commented Jul 30, 2026

Uh oh!

tburt-nv commented Jul 30, 2026

Uh oh!

tensorrt-cicd commented Jul 30, 2026

Uh oh!

tensorrt-cicd commented Jul 31, 2026

Uh oh!

tburt-nv commented Jul 31, 2026

Uh oh!

tensorrt-cicd commented Jul 31, 2026

Uh oh!

tensorrt-cicd commented Jul 31, 2026

Uh oh!

tburt-nv commented Aug 3, 2026

Uh oh!

coderabbitai Bot commented Aug 5, 2026

Uh oh!

tburt-nv commented Aug 5, 2026

Uh oh!

tensorrt-cicd commented Aug 5, 2026

Uh oh!

tensorrt-cicd commented Aug 5, 2026

Uh oh!

coderabbitai Bot commented Aug 5, 2026

Uh oh!

tburt-nv commented Aug 5, 2026

Uh oh!

tensorrt-cicd commented Aug 5, 2026

Uh oh!

tensorrt-cicd commented Aug 6, 2026

Uh oh!

tburt-nv commented Aug 6, 2026

Uh oh!

tensorrt-cicd commented Aug 6, 2026

Uh oh!

tensorrt-cicd commented Aug 6, 2026

tburt-nv commented Jul 28, 2026 •

edited by coderabbitai Bot

Loading