fix(windows): use extended paths in Python bootstraps - #4071
Conversation
|
CI flagged a variety of edge cases: https://buildkite.com/bazel/rules-python-python/builds/17072/list Would you be able to create a repro for this case? i.e. one that triggers LoadLibraryExW with a long path. There's now a py_extension rule that can create windows DLLs, so it should possible to create a small no-op python c module at a long path location to verify it can be loaded I'm gonna throw my ai at this and see if it can hash out the failures, too. |
|
Sorry, I expected to get emails about replies here, but I didn't for some reason (or they were filtered out, etc.), so I missed this. (fixed notifications, they were marked Spam for some reason...) |
|
If you could cross check the current state, that'd be helpful. I threw my agent at it, but haven't vetted what it's done. |
7a8ef71 to
47be690
Compare
Implicit long-path support is not universal across Win32. The documented APIs covered by the long-path opt-in do not include DLL loading functions such as LoadLibraryExW: https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation#functions-without-max_path-restrictions Handle extension paths during import resolution by replacing the standard PathFinder entry in sys.meta_path with a subclass. It only modifies a resolved spec when its loader is ExtensionFileLoader and the extension path reaches MAX_PATH. Add the extended-length prefix only to the concrete extension path, leaving sys.path and sys.prefix unchanged. Convert UNC paths to the \\?\UNC\ form. Add a Windows regression test that loads a real extension module from a path exceeding MAX_PATH without changing other Python paths. Prefixing only sys.path entries that reach MAX_PATH doesn't work: a search root can be shorter than MAX_PATH while the full path of an extension under it may exceed the limit. It would also expose unrelated consumers, such as importlib.metadata and script handling, to extended-path semantics, where relative components such as .. are not normalized.
47be690 to
1ece5f7
Compare
|
I threw an agent at it as well, and it came up with a more filtered solution (explanation in the commit). PTAL Edit: nvm, buildkite doesn't need approval - will check results myself and then let you know |
|
Buildkite is also passing, except one unrelated, non-Windows, at HEAD failure. |
3 participants
Implicit long-path support is not universal across the Win32 API. The documented set of APIs covered by the long-path opt-in does not include DLL loading functions, e.g., LoadLibraryExW:
https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation#functions-without-max_path-restrictions
Always use extended-length paths in Windows bootstrap code and correctly convert UNC paths to the \?\UNC\ form.
Experienced this breakage in JAX after the project being switched over to Bzlmod, which made some paths too long:
https://github.com/jax-ml/jax/actions/runs/31674295160/job/94365413380
Already used as a patch in jax-ml/jax#39961