Summary
toolchain/bootstrap/python.sh serializes uv pip install behind a single lock whose name
depends only on the username:
UV_INSTALL_LOCK="${UV_LOCK_DIR}/mfc-uv-install-${USER:-$(id -un)}.lock"
uv_install() { flock "$UV_INSTALL_LOCK" uv pip install "$@"; }The lock exists to protect the cache, not the venv — the comment above it says so: uv's
own cache lock protects individual entries, but concurrent installs can race while one
extracts/prunes the shared archive-v0 store and leave a corrupted entry.
But the lock is keyed on the user while the thing it protects is the cache directory.
Those coincide today only because UV_CACHE_DIR is redirected only under
GITHUB_ACTIONS=true. For any interactive user, every worktree, checkout and concurrent
session on a machine shares one cache and therefore one lock.
Impact
On a shared login node (Frontier, in my case) two independent MFC sessions in different
checkouts block each other completely. A fresh worktree has an empty build/, so it has no
build/pyproject.toml marker and triggers a full ~93-package install — which takes the
global lock for minutes. Fan out into a few worktrees and everything queues behind one
install.
Worse, the self-heal path amplifies it:
warn "(venv) uv install failed; clearing the uv cache and retrying once..."
uv cache clean
uv cache clean wipes the shared cache. So one interrupted or failed install doesn't
just fail locally — it pulls the cache out from under every other MFC build running on that
machine for that user, causing them to re-download in turn. I hit exactly this cascade: two
concurrent sessions repeatedly re-downloading all 93 packages because each failure cleared
the cache the other was using.
Suggested fix
Key the lock on what it actually guards:
UV_CACHE_KEY=$(printf %s "${UV_CACHE_DIR:-$HOME/.cache/uv}" | md5sum | cut -c1-16)
UV_INSTALL_LOCK="${UV_LOCK_DIR}/mfc-uv-install-${UV_CACHE_KEY}.lock"Installs sharing a cache still serialize (still correct, still protects archive-v0);
installs with distinct UV_CACHE_DIR proceed in parallel safely.
Worth considering alongside it:
- Widen the cache redirect beyond CI. The
GITHUB_ACTIONS-only condition assumes only
CI has concurrency, but interactive users on a shared login node have the identical
problem. Note the tradeoff: node-local caches don't survive across nodes and lose reuse.
- Scope
uv cache clean more narrowly, or drop it from the retry. Wiping a shared
cache as a self-heal for one process's failure is a big hammer with cross-process blast
radius. uv cache prune (removes only unused entries) or clearing just the failed
package would be safer.
Caveat for HPC
Anyone tempted to fix this by setting a per-job UV_CACHE_DIR in a batch script should know
that Frontier compute nodes have no route to PyPI — a fresh cache there fails with
Failed to fetch: https://pypi.org/simple/.... The venv has to be primed on a login node
first. Cost me two dead jobs before I worked that out.
Environment
Frontier (OLCF), cpe/26.03, cce/21.0.2, rocm/7.2.0. Found while investigating
#1684 (moving Frontier off cce/19.0.0).
Summary
toolchain/bootstrap/python.shserializesuv pip installbehind a single lock whose namedepends only on the username:
The lock exists to protect the cache, not the venv — the comment above it says so: uv's
own cache lock protects individual entries, but concurrent installs can race while one
extracts/prunes the shared
archive-v0store and leave a corrupted entry.But the lock is keyed on the user while the thing it protects is the cache directory.
Those coincide today only because
UV_CACHE_DIRis redirected only underGITHUB_ACTIONS=true. For any interactive user, every worktree, checkout and concurrentsession on a machine shares one cache and therefore one lock.
Impact
On a shared login node (Frontier, in my case) two independent MFC sessions in different
checkouts block each other completely. A fresh worktree has an empty
build/, so it has nobuild/pyproject.tomlmarker and triggers a full ~93-package install — which takes theglobal lock for minutes. Fan out into a few worktrees and everything queues behind one
install.
Worse, the self-heal path amplifies it:
warn "(venv) uv install failed; clearing the uv cache and retrying once..." uv cache cleanuv cache cleanwipes the shared cache. So one interrupted or failed install doesn'tjust fail locally — it pulls the cache out from under every other MFC build running on that
machine for that user, causing them to re-download in turn. I hit exactly this cascade: two
concurrent sessions repeatedly re-downloading all 93 packages because each failure cleared
the cache the other was using.
Suggested fix
Key the lock on what it actually guards:
Installs sharing a cache still serialize (still correct, still protects
archive-v0);installs with distinct
UV_CACHE_DIRproceed in parallel safely.Worth considering alongside it:
GITHUB_ACTIONS-only condition assumes onlyCI has concurrency, but interactive users on a shared login node have the identical
problem. Note the tradeoff: node-local caches don't survive across nodes and lose reuse.
uv cache cleanmore narrowly, or drop it from the retry. Wiping a sharedcache as a self-heal for one process's failure is a big hammer with cross-process blast
radius.
uv cache prune(removes only unused entries) or clearing just the failedpackage would be safer.
Caveat for HPC
Anyone tempted to fix this by setting a per-job
UV_CACHE_DIRin a batch script should knowthat Frontier compute nodes have no route to PyPI — a fresh cache there fails with
Failed to fetch: https://pypi.org/simple/.... The venv has to be primed on a login nodefirst. Cost me two dead jobs before I worked that out.
Environment
Frontier (OLCF),
cpe/26.03,cce/21.0.2,rocm/7.2.0. Found while investigating#1684 (moving Frontier off
cce/19.0.0).