lore: Repack the archive repositories as their packs accumulate - #86
Merged
rikvanriel merged 1 commit intoSep 10, 2026
Merged
Conversation
Each fetch of a public-inbox archive writes the objects it received into a new pack. gitoxide implements no repack and, unlike git fetch, never hands off to git gc --auto, so a daily refresh leaves one more pack behind every day. Every object lookup fans out across all of their indices. Roll up the packs that break a geometric size progression, as git repack --geometric does, once an archive holds more than twenty of them. A routine refresh then touches only the small packs recent fetches left. Rewriting the whole archive instead takes minutes on a large list. Skip the .tmp-<pid>-pack-<hash>.idx of a pack a concurrent git is still writing, so the rollup does not delete it, and an index whose pack is missing, since nothing can serve its objects. A .keep file holds a pack out of the rollup while the fetch that wrote it updates the refs. One that outlives that window was left by a writer that died, and its pack rolls up like any other. Remove the multi-pack-index once the old packs are gone. git reports a failed load for each pack a stale one names. The rollup writes its own pack under a .keep and releases it once the old packs are gone. Install gix's signal handler so that a termination signal stops the rollup at a point that still releases the hold, where a kill would leave the new pack out of every later rollup. gix likewise leaves the .keep of a fetched pack in place when no ref changed; drop it after the fetch. Run the rollup after indexing rather than after the fetch, so a large archive does not hold up the emails the caller came for, and only when indexing succeeded, so it does not delay the report of a failure either. Report a failure of the rollup and carry on, since the archive stays correct and only searches more slowly. Pass --no-lore-repack to skip the rollup. Signed-off-by: Chuck Lever <cel@kernel.org>
meta-cla
Bot
added
the
CLA Signed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Each fetch of a public-inbox archive writes the objects it received into a new pack. gitoxide implements no repack and, unlike git fetch, never hands off to git gc --auto, so a daily
semcode-index --lorerefresh leaves one more pack behind every day. Every object lookup then fans out across all of their indices, and the refresh slows down as the packs accumulate.Roll up the packs that break a geometric size progression, as
git repack --geometricdoes, once an archive holds more than twenty of them. A routine refresh then touches only the small packs recent fetches left. Rewriting the whole archive instead takes minutes on a large list.The rollup runs after indexing, so a large archive does not hold up the emails the caller came for, and only when indexing succeeded. A failure of the rollup is reported and indexing carries on, since the archive stays correct and only searches more slowly. Pass
--no-lore-repackto skip it.The rollup runs alongside any git that may be touching the same archive. The commit message describes how it handles a pack a concurrent git is still writing, the
.keepfile that a fetch leaves behind, a termination signal mid-rollup, and a stale multi-pack-index.Unit tests cover the pack selection, the in-flight and held packs, a stale hold, an orphaned index, and the multi-pack-index cleanup, against repositories built in a temporary directory.
Tested by running the daily lore refresh with this change since late August, and against an archive that had accumulated many packs, where the rollup left the expected geometric progression behind.