Skip to content

gpu: fix hipFile probe, cuFile error handling, and driver teardown - #546

Open
potatogim wants to merge 7 commits into
hpc:mainfrom
potatogim:fix/gpu-io-followups
Open

potatogim wants to merge 7 commits into
hpc:mainfrom
potatogim:fix/gpu-io-followups

Conversation

@potatogim

Copy link
Copy Markdown
Contributor

Follow-up fixes for the GPU Direct Storage code merged in #539 and #540. A post-merge review found a configure-time defect, a few smaller correctness issues, and one build failure that shows up with recent CUDA headers.

Problem

  • The hipFile configure probe compiles and links with only the hipFile flags. But hipfile.h includes hip/hip_runtime_api.h, and libhipfile.so can depend on libamdhip64.so without an embedded RUNPATH. With HIP installed under a separate prefix, configure rejects a working hipFile installation before the build even starts.
  • cuFile reads and writes can fail with -1 and errno set (filesystem errors), separate from negative CUfileOpError_t values. The cuFile decoder treated every negative return as an operation error, so the real POSIX cause was lost. The hipFile backend already handled this correctly.
  • Neither backend ever closed its storage driver. POSIX_Finalize was empty, and the cuFile backend's lazy-open flag was never reset. The HIP side relied on implicit initialization during handle registration and had no shutdown path at all.
  • GPU verification could report success without checking data. Allocation, copies, and kernel launches were unchecked, so a runtime failure left the error counter at zero. Verification modes other than timestamps (including incompressible) verified nothing but still returned success.
  • On short GPU transfers the retry loop advanced both the pointer and the buffer offset, so the next call read or wrote at base + 2*r instead of base + r.
  • Compiling src/gpu_io_cufile.c against CUDA 13.3 fails: cufile.h uses loff_t in its callback structs, which glibc only exposes with _GNU_SOURCE. IOR does not enable system extensions, so the cuFile backend does not build on current toolkits.

Change

Seven commits, one per fix:

  • configure: preserve HIP flags in hipFile probes -- the hipFile probe now carries the HIP include flags, library paths, and the detected runtime library in its temporary environment, restoring all three caller variables afterwards. This resolves transitive header and link dependencies for separate HIP and hipFile prefixes.
  • gpu: preserve errno for cuFile transfer failures -- the transfer decoder now distinguishes -1 with errno from negative operation errors, matching the hipFile backend.
  • gpu: close storage drivers after final POSIX instance -- both backends own a lazy-opened driver session and release it when the last POSIX instance finalizes (instance count, since ior dispatches initialize/finalize per test). A failed close is reported as a fatal error rather than ignored.
  • gpu: fail explicitly when verification cannot run -- both verifiers check allocation, copies, launch, execution, and cleanup; the timestamp kernel reports mismatches atomically; unsupported packet types and non-word-sized buffers are rejected instead of silently passing; the common dispatch aborts when verification cannot run rather than letting a failure look like a clean result.
  • posix: keep the GPU transfer buffer base fixed -- GPU paths pass the original buffer base and advance only the buffer offset after short transfers, so completed bytes no longer displace the next transfer twice. Syscall paths are unchanged.
  • configure: enable system extensions for large-file types -- AC_USE_SYSTEM_EXTENSIONS defines _GNU_SOURCE project-wide, fixing the cufile.h loff_t build failure and making the local feature defines in three files redundant (removed).
  • gpu: drop unused variables in the pattern update stub.

Testing

Built on Rocky Linux 10 with ROCm 7.14 and CUDA 13.3:

configuration result
CPU-only (--without-cuda --without-hip, backend none) pass
HIP runtime + hipFile backend pass
CUDA runtime + cuFile backend pass

Carry HIP include flags, library paths, and the detected runtime library into the temporary hipFile probe environment. This resolves transitive dependencies for separate installation prefixes while preserving library order and restoring caller flags.

Signed-off-by: Jihyeon Gim <potatogim@potatogim.net>
Distinguish filesystem failures from negative cuFile operation errors in the shared transfer decoder. Capture errno immediately so callers retain the original cause, while successful and short transfers keep their byte counts.

Signed-off-by: Jihyeon Gim <potatogim@potatogim.net>
Own lazy driver sessions in both storage backends and release them after the last POSIX instance finalizes. Reset ownership only after a successful close and report shutdown failures as fatal errors. Guard lifecycle calls with storage support so CPU and runtime-only builds remain independent.

Signed-off-by: Jihyeon Gim <potatogim@potatogim.net>
Check allocation, copies, kernel launch, execution, and cleanup in both GPU verifiers, and reject unsupported formats and partial words. Use atomic mismatch reporting and skip work for empty timestamp buffers. Abort at common dispatch on verification failure so negative results cannot reach error counters.

Signed-off-by: Jihyeon Gim <potatogim@potatogim.net>
Pass the original buffer base to GPU reads and writes while advancing the buffer offset after short transfers. This prevents completed bytes from displacing the next transfer twice and leaves syscall pointer handling unchanged.

Signed-off-by: Jihyeon Gim <potatogim@potatogim.net>
cufile.h uses loff_t, which glibc exposes with _GNU_SOURCE. Enable system extensions project-wide through the generated config.h, and require them in MPI probes after selecting the MPI compiler.

Remove redundant local GNU feature definitions and use the guarded system-extension template for _XOPEN_SOURCE while retaining its value of 700.

Signed-off-by: Jihyeon Gim <potatogim@potatogim.net>
Remove the unused block and thread counts from the empty CUDA and HIP pattern update functions. Keep both implementations symmetric.

Signed-off-by: Jihyeon Gim <potatogim@potatogim.net>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant