[Gasoonjia]

Builds on #23121 (merged), which ships linkable libraries in the Windows wheel. Stacked on
#23405, which makes the CUDA backend fall back to cudaMalloc on devices without memory
pools: the Windows GPU CI runner's A10G is one, so this PR's end-to-end runs need it.

A Windows user can run a CUDA program but cannot lower one: lowering compiles the model
with a toolchain only the Linux side has, and the CUDA Windows CI job already exports on
Linux and runs on Windows for exactly this reason. What was missing is a wheel that gives
that Windows user the runtime. This adds a Windows CUDA wheel carrying the CUDA delegate
as C++ libraries, the same components the Linux CUDA wheel ships (#21645, #21668), and
nothing CUDA in the Python module, which on Windows has no CUDA program to lower. The
export (AOT) side is unchanged and identical to the CPU wheel's; only the runtime links CUDA.

lib/executorch_backend_cuda.dll    + .lib   the delegate          executorch::backend_cuda
lib/executorch_extension_cuda.dll  + .lib   allocator + streams   executorch::extension_cuda
backends/cuda/aoti_cuda_shims.dll            the AOTI shim layer   (runtime DLL of the delegate)
data/lib/aoti_cuda_shims.lib                 already shipped as the lowering stub; also the
                                             shim layer's import library here
extension/pybindings/_C.pyd                  unchanged: no CUDA dependency, no CudaBackend

A C++ application links it exactly as on Linux:

find_package(executorch REQUIRED COMPONENTS kernels_optimized backend_cuda)
target_link_libraries(app PRIVATE executorch::runtime executorch::kernels_optimized
                                  executorch::backend_cuda)
# plus the $<TARGET_RUNTIME_DLLS:app> copy from the CPU PR, which now also brings the
# CUDA extension and the shim layer

The wheel's DLLs import no CUDA runtime DLL: CMake links cudart.lib into them on Windows,
and that reaches the driver (nvcuda.dll). The compiled model is different: the library
AOTInductor builds into model.pte imports cudart64_13.dll, which comes from the CUDA
Toolkit's bin directory on PATH, as the guide says. PyPI's nvidia-cuda-runtime has win_amd64
builds that carry that DLL, but a C++ program does not search site-packages, so declaring it
would not make it loadable; the wheel declares no NVIDIA package for Windows.

How

• Generator. The CUDA toolkit's Visual Studio integration fails compiler identification
  under the ClangCL toolset on some toolkit and Visual Studio pairs (MSB4023 in its targets
  file), and the MSVC toolset cannot compile the Python extension's sources. A Windows CUDA
  build therefore uses Ninja Multi-Config with clang-cl, the compiler the CPU wheel uses,
  and nvcc with cl.exe as its host compiler, in a Visual Studio developer environment that
  has ninja (ninja, clang-cl and cl all on PATH, which is how the wheel build runs).
  Elsewhere, such as a plain shell, the build keeps the Visual Studio generator it always
  used. CMAKE_CUDA_COMPILER is passed only when CUDACXX is unset, so options a user put in
  CUDACXX (such as -allow-unsupported-compiler) still apply. The toolkit is the one
  install_utils already reports, so what compiles is the train packaging declares.
• CUDA 13.2 and newer. Their CCCL stops with an #error under cl.exe's traditional
  preprocessor, so the CUDA sources pass -Xcompiler=/Zc:preprocessor on Windows (13.0
  accepts it too).
• flatc / flatcc byproducts and imported locations share one host executable suffix chosen
  by CMAKE_HOST_WIN32. CMAKE_EXECUTABLE_SUFFIX is the target's: a Zephyr toolchain sets
  .elf, and naming the byproduct after it left Ninja with no rule for the host tool.
• The shims resolve the runtime from executorch.dll rather than a static core. (The PAL's
  strong symbols are in Ship linkable libraries in the Windows CPU wheel #23121; C++20 for the delegate and shims comes from Fix CUDA backend Windows build under C++20 #23135.)
• The shims keep WINDOWS_EXPORT_ALL_SYMBOLS OFF in a static build, as upstream sets it,
  and turn it ON in the shared layout: the delegate DLL imports the stream guard functions
  (get/set/peek/clearCurrentCUDAStream) from the shim DLL, and those carry no
  AOTI_SHIM_EXPORT, so the annotations alone would not export them. The allocator is not
  in either DLL: main moved it into extension_cuda (Move the CUDA allocator into extension/cuda so other delegates can share it #23447), a shared library the
  delegate and the shims both import it from, so the process has one.
• Ship linkable libraries in the Windows CPU wheel #23121 keeps a Windows CUDA build static; this drops that guard, so the pybind preset
  now builds the shared layout with CUDA too, and adds the CUDA import libraries to the
  Windows packaging list.
• The root CMake file no longer links the CUDA delegate into the Python extension on
  Windows; extension/cuda finds the CUDA toolkit itself, as on main.
• The package config adds the CUDA extension (allocator and stream helpers) and the shim
  layer to the delegate's runtime
  DLL set, and reports EXECUTORCH_RUNTIME_DLLS_EXTRA for the pre-3.28 route, since the
  shim layer ships outside lib/.

CI

• build-wheels-cuda-windows.yml mirrors the Linux CUDA workflow on the Windows wheel
  builder and reuses filter_cuda_matrix.py. pre_build_script.sh drops the blanket Windows
  CUDA=OFF (a CPU row is still forced off by the generic row classification), resolves
  TORCH_CUDA_ARCH_LIST from cuda_arch_list.sh, and writes it to BUILD_ENV_FILE as well as
  GITHUB_ENV: the reusable workflow sources that file after the hook, and its own list had
  replaced the row's (sm_75 in, sm_89 out). It also installs ninja for the generator.
• End-to-end, per published CUDA train, the route a Windows user has, modeled on
  cuda-windows.yml. The trains are read from filter_cuda_matrix.py's SUPPORTED_CUDA_VERSIONS
  (today cu132 and cu134; main stopped publishing cu130 in Stop publishing CUDA 13.0 wheels #23380), the same list the release
  rows are built from, so adding or
  dropping a train there changes what is tested. The end-to-end jobs take their trains from
  the build job's own filtered matrix, so a pull request without the ciflow/binaries/all
  label tests the one train it builds, and a train the generator dropped is not tested
  against a wheel that does not exist. (install_utils.py's list also has 12.6 for
  source builds, but 12.6 is no longer a published wheel train.) A Linux GPU job exports a
  Windows-target program with that train's torch (test_cuda_windows.py --export); a small
  job bundles it with the wheel the build job published for that train (windows_job.yml
  downloads one artifact and gives its script no token for a second); and a Windows GPU job
  installs that published wheel and runs test_cuda_windows.py with the program
  (test_cuda_windows_e2e.ps1), so CI runs a CUDA program through the artifact users install
  rather than a wheel rebuilt on the GPU runner. The export and end-to-end jobs check out the
  commit the build job builds (on a pull request, its merge commit), so the wheel, the
  program and the scripts that test them match. Each train's CUDA (on Linux for the export,
  and on Windows for the smoke test's cuobjdump and the model's CUDA runtime) is assembled
  from NVIDIA's checksummed redistributable archives by
  install_cuda_redist.py, which needs no installer or admin rights and fails on a train it
  has no release for. Each train's Windows job waits only for its own bundle, so one train
  failing does not skip the others, and like the export it does not run for a
  pull request from a fork. The Windows GPU runners come without an NVIDIA
  driver (nvcuda.dll is missing and cudaGetDeviceCount returns cudaErrorInsufficientDriver),
  so the job first installs the same data-center driver, from the same place, that
  pytorch/pytorch's Windows CUDA smoke tests install (driver_update.bat, 580.88); cu132 and
  cu134 run on it through CUDA's minor-version compatibility, as PyTorch's own wheels do. The
  wheel build's own smoke test has no GPU program and says so.

Tests

test_cuda_windows.py, the Windows counterpart of test_cuda_linux.py:

• the CUDA DLLs and import libraries ship;
• _C depends on no CUDA DLL and registers no CUDA backend;
• no shipped CUDA DLL imports cudart64_*.dll (only the driver), and no Windows NVIDIA
  requirement is declared;
• every DLL with device code carries exactly the row's GPUs, both directions, and the
  newest also as PTX; the expected list comes from executorch_cuda_arch_list in
  cuda_arch_list.sh, called the way the pre-build hook calls it and keyed by the wheel's
  +cuXYZ tag, not from the TORCH_CUDA_ARCH_LIST the build consumed;
• a C++ application linking executorch::backend_cuda gets the three DLLs copied beside it
  and sees CudaBackend registered;
• a program lowered on Linux for Windows runs through the C++ SDK and matches eager, after
  checking it was exported with the same CUDA train as the wheel (every CUDA 13 train loads
  cudart64_13.dll, so a mismatch would not fail on its own);
• then what a CPU Windows row checks: test_shared_libraries.py (with the three CUDA
  ownership rows), test_cpp_sdk.py and the MobileNetV3/XNNPACK model run, including the Arm
  Cortex-M Python module, which builds under Ninja too: CMSIS-NN's test-only shared library
  took the static library's name, so its import library collided with it on Windows, and
  backends/cortex_m now gives the import library its own name.

Test plan

On Windows 11 x64 with an RTX 5080 (sm_120, driver 595.95), VS 2022 BuildTools, CPython 3.12,
and WSL Ubuntu for the export, once per CUDA train. 13.2 and 13.4 are the published trains
and the ones CI covers; 13.0 is no longer published (#23380) but still builds from source, so
it was run too:

train	Windows nvcc	WSL export torch	wheel	test_cuda_windows.py	Linux-exported program
13.0	13.0.88 (installer)	2.14.1+cu130	executorch-1.6.0+cu130	52 checks pass	runs on the GPU, maxdiff 2.4e-7
13.2	13.2.86 (redist)	2.14.1+cu132	executorch-1.6.0+cu132	52 checks pass	runs on the GPU, maxdiff 2.4e-7
13.4	13.4.59 (redist)	2.14.0.dev20260810+cu134	executorch-1.6.0+cu134	52 checks pass	runs on the GPU, maxdiff 2.4e-7

• Each wheel built with TORCH_CUDA_ARCH_LIST from cuda_arch_list.sh: device code for exactly
  sm_80/86/89/90/100/120, plus sm_120 PTX. 13.4 runs on a driver that reports 13.2, through
  CUDA's minor-version compatibility.
• Before the /Zc:preprocessor change, 13.2 and 13.4 failed to compile the .cu shims with
  CCCL's "traditional preprocessor" #error; 13.0 was unaffected.
• Negative: the cu134 wheel pointed at the 13.0-exported program fails the train check.
• The export used test_cuda_windows.py --export from this branch in WSL, with mingw-w64 and
  the Windows CUDA runtime of the same train (13.0 from install_cuda_windows_cross_compile.sh,
  13.2/13.4 from the redist archives, the steps the guide now gives).
• CI, before the rebase onto main: the end-to-end jobs passed for both published trains on
  the Windows A10G runner, with a Linux-exported program matching eager (maxdiff 1.3e-7 for
  13.2, 2.4e-7 for 13.4).
• After rebasing onto main with the allocator moved into extension_cuda (Move the CUDA allocator into extension/cuda so other delegates can share it #23447), the cu132
  wheel was rebuilt and passes all 55 checks, including the new ownership row: a single CUDA
  allocator and stream helpers owned by executorch_extension_cuda.dll across the 15 shipped
  libraries. The rows above were measured before that rebase.
• Linux: the CMake changes are behind WIN32 / MSVC, or pick the same suffix there (empty on a
  Linux host); setup.py's new path runs only on Windows.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/23123

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 4 New Failures, 1 Cancelled Job, 1 Pending, 3 Unrelated Failures, 1 Unclassified Failure

As of commit fd104a7 with merge base aaa31ad ():

NEW FAILURES - The following jobs have failed:

• Apple / build-benchmark-app / macos-job (gh)
• Apple / build-demo-ios / macos-job (gh)
• trunk / test-qnn-optimum-model (fp32, distilbert) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• trunk / test-torchao-huggingface-checkpoints (lfm2_5_1_2b, mt-l-arm64g4-16-62, executorch-ubuntu-22.04-gc... / linux-job (gh)
  ##[error]error: RPC failed; curl 56 GnuTLS recv error (-54): Error in the pull function.

UNCLASSIFIED FAILURE - DrCI could not classify the following job because the workflow did not run on the merge base. The failure may be pre-existing on trunk or introduced by this PR:

• Build Windows CUDA Wheels / test-cuda-windows-wheel-e2e-13.2 / windows-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  AssertionError: no built wheel was found to inspect, so the platform tag was never checked. Run this from a tree where the wheel was built, or set EXECUTORCH_TEST_WITHOUT_WHEEL to state that no wheel is expected

CANCELLED JOB - The following job was cancelled. Please retry:

• pull / unittest / windows / windows-job (gh)

FLAKY - The following job failed but was likely due to flakiness present on trunk:

• pull / unittest-editable / windows / windows-job (gh) (matched win rule in flaky-rules.json)
  ##[error]The operation was canceled.

BROKEN TRUNK - The following job failed but was present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Apple / build-frameworks-ios / macos-job (gh) (trunk failure)
  /Users/runner/work/executorch/executorch/pytorch/executorch/pytorch/aten/src/ATen/mps/MPSDevice.mm:129:73: error: use of undeclared identifier 'MTLLanguageVersion3_2'

UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:

• trunk / test-samsung-models-linux / linux-job (gh) (#23507)
  test_yolo26_fp16

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[shoumikhin]

Some notes inline.

[shoumikhin]

I think this find_package(CUDAToolkit) is no longer needed. Since an earlier fix for exactly this case, extension/cuda finds the toolkit itself right before it links CUDA::cudart. I tried a small project with that layout and no Torch, and it configured and built without a find in the parent. The comment describes the old state. The block also runs on Linux, while the test plan says the CMake changes are behind WIN32. Could you drop these lines?

[Gasoonjia]

Agreed, removed; the top-level CMakeLists.txt is back to main around that block. extension/cuda finds the toolkit itself before linking CUDA::cudart, and the rebuilt cu132 wheel configures and builds without it.

[shoumikhin]

This check and the end to end PowerShell script each parse cuda_arch_list.sh with a regex. The Linux check, and the Windows branch of the pre-build hook in this pull request, source the script and call executorch_cuda_arch_list. The regex only understands a plain string or one variable reference, so a reference plus one more architecture gives a wrong list. bash is on the Windows runners, so could both call the function the same way? Then there is one source of truth.

[Gasoonjia]

Done. Both now call executorch_cuda_arch_list through bash, the way the pre-build hook does, so cuda_arch_list.sh is the single source of truth. They prefer Git's bash (System32's bash.exe is WSL's launcher, which can come first on PATH), pass the train on the command line, and strip carriage returns while sourcing, so a checkout with core.autocrlf works too. Checked locally: cu132 and cu134 resolve to the same list as before, and an unknown train fails with the script's own error.

[shoumikhin]

The docstring says every DLL with device code also carries the newest architecture as PTX, and the description says the same. The check only needs one DLL to have it, because assert portable passes as soon as the list is not empty. With a second library that has device code but no PTX, the test still passes and its message says the PTX is there. Could it assert that no library with device code is missing the PTX?

[Gasoonjia]

Right, it only required one. It now asserts that no library with device code is missing the newest architecture as PTX, and names the ones that are.

[shoumikhin]

Some notes inline.

[shoumikhin]

The matrix comes from SUPPORTED_CUDA_VERSIONS, which no longer has 13.0. So this check is always true, and the image toolkit path never runs. I would drop the condition and always use the redist archives. The comment on the table in install_cuda_redist.py has the same problem: it says one entry per published train, but 13.0 is there too.

[Gasoonjia]

Agreed: dropped the condition, so every train uses the redist archives, and updated the image comment. The _RELEASES comment now says why 13.0 stays there: it is no longer published but still builds from source, and local tests of it use that entry.

[shoumikhin]

Two things from this round, plus some stale text.

The description still says the root CMake file finds the CUDA toolkit first, because extension/cuda links CUDA::cudart before backends/cuda finds it. That find was removed during review, and the root file now only adds the Windows guard, so that bullet no longer matches the code.

In setup.py, the comment on the CUDA runtime requirements says a CUDA wheel is only built on Linux. Windows builds one now, so it would help if that comment said why Windows still needs no NVIDIA package.

One process thing: without the ciflow/binaries/all label this builds and runs only the 13.2 train, for one Python version. Worth adding the label once before merge so 13.4 is checked on the final code.

[shoumikhin]

This job builds its own wheel from source on the GPU runner, so the wheel we publish is never the one that runs a CUDA program.

The build job's own smoke test skips that run: it needs EXECUTORCH_CUDA_WINDOWS_ARTIFACTS, which is not set there, so it prints a skip and moves on. Between the two jobs, nothing in CI runs a program through the released artifact, and a problem that only shows up at run time in the release build would pass.

Could this job download the wheel the build job made for the same CUDA version and install that instead? Then CI tests what users install, and each train saves a full wheel build on a GPU runner.

[Gasoonjia]

Done. The end-to-end job no longer builds a wheel. A small bundle-cuda-windows-wheel-e2e job (one per train) downloads the wheel the build job published for that train and the exported program, and uploads them as one artifact, because windows_job.yml downloads a single artifact and gives its script no token for a second. The Windows job installs that published wheel (with a conda env matching its cpXY tag) and runs the program through it, so CI now tests what users install, and each train saves a wheel build on a GPU runner.

Two things came with it. The trains come from the build job's own filtered matrix, so they always match the wheels that exist, including when the generator drops a train or a limited PR falls back. And the export and end-to-end jobs check out github.sha, the commit the build job's empty ref builds (on a PR, its merge commit), so the wheel, the program and the test scripts come from one commit. I ran the new script locally against my cu132 wheel and a WSL-exported program: it installs the wheel and the full smoke test passes, with the program matching eager. The bundle step itself only runs in CI.

[shoumikhin]

This comment says MSVC links the delegate into portable_lib. The root file now skips aoti_cuda_backend on Windows entirely, so after this change no Windows build does that, and the reason given for the allocator copy reads wrong.

The copy still looks needed, for C++ applications that link the static delegate and cannot import the allocator from the shims DLL in this layout. Could the comment say that instead?

[Gasoonjia]

Right, that reason is gone. The comment now says why the copy is still there: in the static layout a C++ application links the delegate as an archive and, on MSVC, cannot import the allocator's C++ symbols from aoti_cuda_shims.dll.

[Gasoonjia]

Thanks. On the review-level notes:

• Description: the bullet about the root CMake file finding the toolkit is gone. It now says the root only keeps the CUDA delegate out of the Python extension on Windows, and extension/cuda finds the toolkit itself, as on main.
• setup.py: the comment on the CUDA runtime requirements now says why Windows needs no NVIDIA package: the shipped DLLs load only the driver, and the CUDA runtime a model's own library imports comes from the CUDA Toolkit install.
• ciflow/binaries/all: agreed, I'll add it once before merge so 13.4 runs on the final code.