Regression stability + new regression tests

[scarlehoff]

• Add a test for vp-setupfit
• Improve the stability of the regression tests
• Include a theory covmat test
• Generate the data in a dedicated worker
• Set the tolerance with respect to said worker

Fixes #2464

[Radonirinaunimi]

I'm afraid self-hosting is not a possibility, the difference is too big and I don't think we have the infrastructure to run the regression tests on the self-hosted runner for every commit :(

Let's see whether I can find a tolerance value for which is acceptable and for which the regressions don't need to be redone that often.

Why don't we then use a container? This would allow use to have infinite reproducibility?

[scarlehoff]

We are effectively running in a container. I haven't tested, but if the differences are on the underlying hardware, the container won't fix it.

[Radonirinaunimi]

But is it really at the level of the hardware though? Not at the level of system packages, compilers, etc.?

[scarlehoff]

Well, unless the ubuntu-latest means different things, yes. If you get two logs from two workers that generate the differences the only difference is the AWS region in which they are running (I don't have enough statistics to know whether it would happen in the same region or whether it would work in different regions though, but it's what I have... so the only way around is a tolerance...)

[Radonirinaunimi]

Selecting ubuntu-latest for example only ensures OS upgrades but on top of this, GitHub continuously updates the runner images themselves (system packages, compiler, tools, etc.). And I am pretty sure these are the causes of the non-reproducibility. And this is really the main reasons (on top of LHAPDF) we run regressions in container in PineAPPL and NeoPDF for example. I could try to give a look whenever I have some time.

[scarlehoff]

The non-reproducibility here happens non deterministically though. But indeed, it could be updates that made one azure region be different from the other

[scarlehoff]

Ok, for the time being generating the reference in a custom worker and defining the tolerances with respect to it seems to work. I can try to make it run a few more times to get more statistics.

@Radonirinaunimi if you have time to hace a quick look at this it would be very much appreciated. I'm not very happy with the tolerances and if that can be fixed with a docker image please do so. But at the moment I'm more worried about having regressions that we can rely on (so that people don't learn to simply ignore the X).