Unexpected triggering of multithreading

[thijsjanzen]

When submitting R packages for verification to CRAN, we receive (on multiple packages), the following results:

* checking tests ... [114s/21s] NOTE
  Running ‘testthat.R’ [114s/21s]
Running R code in ‘testthat.R’ had CPU time 5.6 times elapsed time
* checking for unstated dependencies in vignettes ... OK
* checking package vignettes ... OK
* checking re-building of vignette outputs ... [165s/26s] NOTE
Re-building vignettes had CPU time 6.4 times elapsed time

Suggesting that the code was executed in a multi-threaded fashion. However, in all our code-calls, we set the number of threads to 1. We have double checked this, and now also return the number of threads after executing back to 1. To no avail.

Our packages use tbb in Rcpp, and we set the number of threads on the R side through:

RcppParallel::setThreadOptions(num_threads)

And collect the number of threads on the Rcpp side through:

auto* nt_env = std::getenv("RCPP_PARALLEL_NUM_THREADS");
auto num_threads = (nullptr == nt_env)
                                ? tbb::task_arena::automatic
                                : static_cast<size_t>(std::atoi(nt_env));
auto global_control =  tbb::global_control(tbb::global_control::max_allowed_parallelism, 
                                                                        num_threads);

Parallelization inside the Rcpp code is then invoked through tbb::parallel_for .

It seems however that at the check at CRAN (Debian, Clang 20), the global control doesn't work, and multiple threads are still created.

I can not replicate this issue on macos (my machine) or on rhub (no multi threading) or GHA (no multi threading).

Is this a known issue // are we setting something up incorrectly somewhere?

[eddelbuettel]

Is this a known issue //

Sort of.

are we setting something up incorrectly somewhere?

Likely.

This can be frustrating, and there are different ways to go about it as there are also different environment variables. See for example the (older, but still very helpful) package RhpcBLASctl.

One time I reworked a package that way I ensured each example and test function started with a blunt brute-force 'collect current thread count, force it to two to pass CRAN' and I think I may have relied on reset via on.exit(). You could do that there to muffle example and tests (and add it a vignette start) leaving your core performance multi-threaded.

I also recently cleaned this up for RcppArmadillo in the opposite way. If you have the common variable CRAN listen set locally to your proper value then you get the performance, and CRAN gets the dialed-down one. Everybody is happy.

> library(RcppArmadillo)
RcppArmadillo 15.2.6-1 using 32 cores. See 'help("RcppArmadillo-package")' for details.
> 

So without seeing more / without a minimally replicating example it is hard to know what is going on.

PS This has also been discussed quite a bit on lists over the years. Likely r-package-devel, maybe also rcpp-devel.

[eddelbuettel]

Maybe RcppParallel::setThreadOptions( getOption("Ncpus", RcppParallel::defaultNumThreads()) ) can help? It would restrict it down on CRAN (as they set Ncpus as an option field and expect you to honour it) with the hardware default as a fallback otherwise?

[thijsjanzen]

Yeah, I realize this is a tough nut to crack / provide feedback on, as I don't have a minimal example, or working test bench - currently, the only way to test is by submitting to CRAN, which seems like a weird route.

Both using RhpcBLASctl and forcing the getOption("Ncpus") at the start of the vignettes did not work, and computation time remained unaffected. It thus seems like all entry points into multithreaded code correctly set the number of threads (or at least do so consistently), as pre-forcing the number of threads before execution of a vignette did not change computation speed.

Currently, at all entry points, the number of threads is set to 1, but somehow in Debian this is not respected it seems. I guess this remains hard to debug, and I'll first work on a minimal example in a VM to further this.

[eddelbuettel]

Sorry to be breaker of bad news here but I would recommend to step back. This problem is as old as parallel code use it at CRAN, i.e. decades, and all other packages appear to cope. Yours is not. So I would think you do something different.

The recommendation in Writing R Extensions is to honour Ncpus and set resources explicitly. That is what I do e.g. in RcppArmadillo: If I weren't, we'd get all cores taken each time RcppArmadillo is loaded. Obviously a violation. So we avoid it.

RcppParallel offers you entry routes. I think without a minimally reproducible example I can only suggest you throttle on entry on systems with working OpenMP setup. (That is generally Linux, but there is no Debian blame here as the Fedora systems at CRAN behave (in this aspect) like the Debian machines at CRAN.)

[eddelbuettel]

Is is a little awkward that I have to guess this, but are we talking about junctions? Or GenomeAdmixR?

Ok by manually searching over your packages and repos (thanks for putting links to GH repos in DESCRIPTION) I can trace this to e.g. this commit. Which looks good to me. If it does not bite, then maybe parallel code is entered / triggered elsewhere too?

That is a large package with a lot of dependencies. On my (many cores) machines I can see

[...]
✔  checking for sufficient/correct file permissions ...
─  checking whether package ‘GenomeAdmixR’ can be installed ... [34s/4s] NOTE (4.4s)
   Installation took CPU time 7.7 times elapsed time
─  used C++ compiler: ‘g++-15 (Ubuntu 15.2.0-4ubuntu4) 15.2.0’
─  checking C++ specification ... INFO
     specified C++17
✔  checking installed package size ...
[...]
✔  checking for unstated dependencies in ‘tests’ ...
─  checking tests ...
   Running R code in ‘testthat.R’ had CPU time 4.3 times elapsed time
    [50s/11s] NOTE
[...]
✔  checking files in ‘vignettes’ ...
─  checking examples ... [5s/2s] OK (2.2s)
✔  checking for unstated dependencies in ‘tests’ ...
─  checking tests ...
   Running R code in ‘testthat.R’ had CPU time 4.3 times elapsed time
    [50s/11s] NOTE
   * checking for unstated dependencies in vignettes ... OK
   * checking package vignettes ... OK
   * checking re-building of vignette outputs ... [102s/13s] OK
   * checking PDF version of manual ... OK

and I guess I did not turn the option for 'warn for vignettes too' but [102s/13s] is clear.

Setting OMP_THREAD_LIMIT=2 did not help. Also setting OMP_NUM_THREADS=2 appears to not help either. (I fell into a trap when I stopped my console from scrolling.)

[thijsjanzen]

Is is a little awkward that I have to guess this, but are we talking about junctions? Or GenomeAdmixR?

Thanks a lot for thinking along!

It is GenomeAdmixR (which, admittedly, I would currently code a bit more soundly), but also secsse, which displays the same behaviour upon submission to CRAN.

I'm currently trying to identify / fix the issue in GenomeAdmixR, as the setup there is similar to secsse (of which I'm not the author, only contributor). Overall, both these packages use tbb::parallel_for (not OpenMP), and set the number of threads through RcppParallel::setThreadOptions, and then collect the number of threads on the Rcpp side again, so I expect fixing one package should also provide insight in fixing the other package.

Being able to reproduce the issue on a VM would still help me a lot I think, as currently, trying solutions by submitting new versions to CRAN in order make them run on a multi-core Debian system seems... wasteful.

I do think that we set the resources explicitly, but we do indeed not honour Ncpus explicitly, this could be improved.

[eddelbuettel]

See my comment (which I edited) posted before yours. In short, it's messy, but it is a big package doing lots of things and one entry point is not your friend. I fear you may have to find that by trial and error.

My advice, and this may sound like a detour, is to set up a small package containing only the package you suspect causes this, and demonstrate to yourself you can control it. Then add other packages. If all goes well at the end when you have all your dependencies back you still behave under the CPU limit.

The fact that you do not have the issue with junctions may rule out some packages, or maybe you do things differently there. The discussion in WRE is not all that helpful either, sadly. As I said above, there have been prior discussions on the mailing lists and some people (IIRC Henrik among them) suggested better global control from R. We do not have that, and you are left chasing a very loose end. Sorry about that, but I cannot suggest a better fix either.

[eddelbuettel]

Being able to reproduce the issue on a VM would still help me a lot I think

You can tell VMs and Docker to use more / all cores. That may be a matter of what host operating system you have. If you only have macOS or Windows around you maybe you should consider renting a cheap small Hetzner instance. It will repro this. You also seemingly get four cores at GitHub Actions so you could use that. Be mindful of what test runner / actions framework you use. An action entry of

      - name: Show nproc
        run: nproc

in a minimal workflow file running ubuntu-latest (using a test repo I use for my r-ci setup relying for Linux on r2u) gets me only two which is odd

and in contradiction to what I quoted above. Oh well!

PS Ah, documented further below: Private repos only get two. Confirmed!

[thijsjanzen]

Ok, I "thought" that the issue only arose on Debian, but on my macbook, it also happens, CRAN check just doesn't generate a NOTE (the specific ratio between executed and user time is hardware dependent, and apparently less critical on my macbook).

Having made a minimal repro and playing around with the code, it seems that embedding the parallel_for within a task_arena fixes the issue.
Knowing this, the documentation of tbb::task_arena does mention that this is "the way" to control thread usage. Hindsight is 20/20...

Next to submit to CRAN and see if on their system, the issue now also disappears.

[eddelbuettel]

Ah, yes, that is a 'CRAN special': The test happens on incoming and on r-devel for both windows and linux but is not (in general) OS or (worse !!) distro-specific. It takes more effort to make macOS use multiple cores but if one does (and e.g. makes OpenMP available, or alternatives such TBB, or ...) then of course the same issue arises. I think we can close this then.

[eddelbuettel]

Knowing this, the documentation of tbb::task_arena does mention that this is "the way" to control thread usage. Hindsight is 20/20...

Should we document this better? The Rcpp Gallery is still there and quiet, if we slap a bit of markdown around an example it would at least be "out there..."

[kevinushey]

I suppose a somewhat related question here is -- IIRC, RcppParallel manages this for you if you use the RcppParallel APIs like parallelFor and parallelReduce (that is, we set up the arena etc. for you). Is using these an option? Or is there something else that RcppParallel that could provide that would make reaching directly for TBB unnecessary?

[eddelbuettel]

Excellent follow-up. I also noticed that @thijsjanzen was doing mano-a-mano combat with TBB on musl. Ex ante the package handles all that. But if one takes the package as a mere vessel for transporting header files ... then one ends up reinventing a lot of the things that went into the package for all those years.

[thijsjanzen]

Excellent follow-up. I also noticed that @thijsjanzen was doing mano-a-mano combat with TBB on musl. Ex ante the package handles all that. But if one takes the package as a mere vessel for transporting header files ... then one ends up reinventing a lot of the things that went into the package for all those years.

Yes, it seems from our side we have completely overlooked the pre-existing wrappers within the package. Coming from a pure C++ implementation, and then moving towards an R/Rcpp/RcppParallel integration, I fear we have mainly used RcppParallel as a source for the TBB headers. This is perhaps better visible in the secsse package: (example of such code: https://github.com/rsetienne/secsse/blob/master/src/secsse_loglik.h )

I also don't think this is all unclearly documented on the RcppParallel side per se, but more from the intel::TBB side perhaps.