database: treat database as read-only after creation and use shared locks

[fischeti]

In #307 we introduced database locks and separaring database from checkout in order to share the database as a cache cross projects. This is also relevant in the context of CI, since the git databases are only fetched once and can then be reused. However, the exclusive locks on the databases essentially prevent any kind of parallelization and every invocation of bender becomes serialized across CI jobs.

This PR aims to relax the locks to use exclusive locks only when writing to the database, and use shared locks when reading from it.

The flow of fetching and checking out a git repository was slightly adapted to reduce the amount of write operations on a shared database:

• git init and git fetch [<rev>] are write operations and still acquire an exclusive lock
• The creation of a temporary tag tmp-<hash> in the database and the later checkout with git clone --branch tmp-<hash> has been replaced with git clone --shared --no-checkout && git checkout <rev>
• The database is configured to disable git gc since revisions might not be tracked anymore by refs i.e. the previous tmp-<hash> tags. This is maybe overly catious.

Furthermore, file locking is done more fine grained. Instead of locking the whole git_database call exclusively, only the part that requires writes (git init and git fetch) acquire an exclusive lock (if database is not ready yet).

[fischeti]

Actually, I am not sure anymore if the locks are needed. I have the impression that git should be able to handle concurrency itself of a shared git database i.e. objects are immutable and do not need to be locked, while other things like refs seem to have fine-grained lock files:

https://stackoverflow.com/questions/19962024/locking-strategy-of-git-to-achieve-concurrency
https://stackoverflow.com/questions/750765/concurrency-in-a-git-repo-on-a-network-shared-folder#answer-751026

Edit: I guess the git init cannot be locked properly by git and could cause problems and apparently git also errors out when failing to acquire locks, so it could make sense after all🤓

[micprog]

Actually, I am not sure anymore if the locks are needed. I have the impression that git should be able to handle concurrency itself of a shared git database i.e. objects are immutable and do not need to be locked, while other things like refs seem to have fine-grained lock files:

https://stackoverflow.com/questions/19962024/locking-strategy-of-git-to-achieve-concurrency https://stackoverflow.com/questions/750765/concurrency-in-a-git-repo-on-a-network-shared-folder#answer-751026

I think we should preserve locks, and I like your solution to speed things up (still need to review the details). The main issue I'm attempting to solve is with multiple repos fetching the same bare ref. Concurrent fetches racing on the same ref would produce fatal: cannot lock ref ...: File exists from one of them. Git handles it cleanly but bender has no retry layer, so we'd just propagate that as a CI failure. Rare, but the failure mode is "this CI job randomly fails once a week". Keeping locks fixes this.

[fischeti]

Yes, you are right🤓 I just realized that git locks are not blocking but just error out