Default number of CPUs chosen looks at available memory in a way that doesn't really make sense

[KristofferC]

On my M4 I get:

julia> using ParallelTestRunner

julia> ParallelTestRunner.default_njobs()
3

Running tests on this CPU with max 3 cores is very wasteful. The reason is some memory limitation at https://github.com/JuliaGPU/CUDA.jl/blob/c5145ab2043cca295b05e840fb2fc632797f1b9c/test/runtests.jl#L175 but the numbers there seem very arbitrary. I'd probably skip the memory consideration completely when chosing how many cores to use.

[giordano]

Basically duplicate of #3

[giordano]

At some point I tried #65 to be able to have a custom default number of jobs (main use case being CI, on GitHub Actions where macOS runners have little memory you get only one job out-of-the-box, which is meh), but decided not to continue.

[KristofferC]

Basically duplicate of #3

I think not. That seems to be about computing the available memory accurately. This is about the choice of the fixed "cost" in the memory of a worker which I am not sure should be used at all. Are there any systems with n cores that cannot run n julias?

[vchuravy]

We should use

ParallelTestRunner.jl/src/ParallelTestRunner.jl

Line 36 in 19cd147

const max_worker_rss = JULIA_TEST_MAXRSS_MB * 2^20

as the constant and then add a environment variable to set this. That number comes from the Julia tests and is the high water mark at which point we will restart a worker

[giordano]

Are there any systems with n cores that cannot run n julias?

Yes, a64fx is one of them, there are only 32GB of total memory for 48 cores, just spawning 48 julia processes would use about half of the total memory. And also the GitHub-hosted aarch64 macOS Actions runners have relatively little memory, 7 GB for 3 vCPUs, and this system quite relevant to this package, I'm mildly sure Enzyme tests would run out-of-memory if we used all 3 cores (but the default of 1 is also suboptimal, so we're forcing 2: EnzymeAD/Enzyme.jl#2690)

[KristofferC]

I'm mildly sure Enzyme tests would run out-of-memory

I thought this package was a general testing package and not only for Enzyme. Obviously concrete packages can use unbounded memory and require fewer CPUs to be used.

[giordano]

I thought this package was a general testing package and not only for Enzyme.

Of course it's not an Enzyme-exclusive, but we're trying to say that memory is a factor in determining the number of processes, which you're trying to dismiss entirely. #3 was closed, but it wasn't really addressed the ability to change the memory requirement, which is still hard-coded at the moment, and #73 (comment) suggested to change that.

[KristofferC]

but we're trying to say that memory is a factor in determining the number of processes, which you're trying to dismiss entirely

I am not dismissing that, I am saying that that choice has to come from a user. Regarding #73 (comment), it mentions max_worker_rss, which is 3.7 GB, which is when the process will restart. I don't understand why that should determine how many cores I want to run with?

[KristofferC]

Just an example from Ferrite-FEM/Ferrite.jl#1370

┌───────────────────────────┬────────┬───────────────┬───────────────┬───────────────┐
│         Test job          │ master │ PR @deb448c18 │ PR @a5b1c39c1 │  Δ vs master  │
├───────────────────────────┼────────┼───────────────┼───────────────┼───────────────┤
│ Julia 1 - windows-latest  │ 19m47s │ 11m06s        │ 10m09s (609s) │ −9m38s (−49%) │
├───────────────────────────┼────────┼───────────────┼───────────────┼───────────────┤
│ Julia 1 - ubuntu-latest   │ 19m15s │ 14m29s        │ 11m00s (660s) │ −8m15s (−43%) │
├───────────────────────────┼────────┼───────────────┼───────────────┼───────────────┤
│ Julia 1 - macOS-latest    │ 15m44s │ 12m18s        │ 7m56s (476s)  │ −7m48s (−50%) │
├───────────────────────────┼────────┼───────────────┼───────────────┼───────────────┤
│ Julia pre - ubuntu-latest │ 15m07s │ 8m11s         │ 7m46s (466s)  │ −7m21s (−49%) │
├───────────────────────────┼────────┼───────────────┼───────────────┼───────────────┤
│ Julia lts - ubuntu-latest │ 12m12s │ 7m13s         │ 6m51s (411s)  │ −5m21s (−44%) │
└───────────────────────────┴────────┴───────────────┴───────────────┴───────────────┘

The a5b1c39c1 commit is by forcing Sys.CPU_THREADS jobs. You can see what a big impact it has from the default on e.g. mac.