indexserver: reuse shallow clones for opted-in repositories - #1162
Merged
Merged
Conversation
Keep Git acquisition and cleanup out of gitIndex so it only orchestrates configuration and indexing. Group the existing Git operations under gitRepoCache and lend the fetched directory through a callback. This deliberately preserves temporary-clone behavior, including Git commands, delta fallback, and unconditional best-effort deletion. Keeping this separate from retention policy makes the upcoming monorepo cache change reviewable. Amp-Thread-ID: https://ampcode.com/threads/T-01a0717a-3acd-7026-977e-6ffe6d30e027 Co-authored-by: Amp <amp@ampcode.com>
…e reuse Allow Sourcegraph to opt selected repositories into retaining shallow clones between indexing jobs. Git GC retains unreachable promisor objects, so expire clones after seven days rather than allowing frequently indexed monorepos to grow indefinitely. The existing tmp-root cleanup still clears them on restart. Keep reuse eligibility and completion metadata inside gitRepoCache. Failed or interrupted jobs, source/filter changes, opt-outs, unassignment, and tenant deletion must not leave stale clones reusable. This acquisition policy does not affect search-index identity or the gitIndex callback. Integration coverage verifies object reuse, updated searchable revisions, filter expansion, delta fallback, and failure recovery. Lifecycle tests also cover temporary clones, callback panics, and completion-marker failures. Amp-Thread-ID: https://ampcode.com/threads/T-01a0717a-3acd-7026-977e-6ffe6d30e027 Co-authored-by: Amp <amp@ampcode.com>
keegancsmith
force-pushed
the
k/cache-git-repo
branch
from
September 5, 2026 13:21
1389710 to
7e340fd
Compare
We can just always set atomic and no-auto-maintenance. Additionally we can always do replaceall for the header config
keegancsmith
changed the title
keegancsmith
marked this pull request as ready for review
September 5, 2026 13:42
stefanhengl
reviewed
Sep 7, 2026
| } | ||
|
|
||
| func (gitRepoCache) fetch(ctx context.Context, gitDir string, o *indexArgs, c gitIndexConfig, logger sglog.Logger) error { | ||
| if err := initGitRepo(ctx, gitDir, o, c); err != nil { |
Member
There was a problem hiding this comment.
will this error if we call it twice?
Member
There was a problem hiding this comment.
never mind, it gets a fresh tmp dir
stefanhengl
approved these changes
Sep 7, 2026
stefanhengl
reviewed
Sep 7, 2026
|
|
||
| var gitCache gitRepoCache | ||
|
|
||
| const gitRepoCacheMaxAge = 7 * 24 * time.Hour |
Member
There was a problem hiding this comment.
Is it worth making this tunable?
Member
Author
There was a problem hiding this comment.
Agreed will add envvar SRC_GIT_REPO_CACHE_MAX_AGE before merging
A fixed seven-day lifetime is a sensible default, but deployments may need a faster disk-pressure escape hatch or longer reuse window. Follow the existing indexserver duration environment pattern so operators can adjust the policy without expanding the per-repository protocol. Amp-Thread-ID: https://ampcode.com/threads/T-01a07b93-fe3d-7234-b192-79dcabb36f90 Co-authored-by: Amp <amp@ampcode.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Allow Sourcegraph to request
CacheGitReposo repeated monorepo indexing reuses existing Git objects. Clones expire after seven days or a restart; failed jobs, source/filter changes, opt-outs, and tenant deletion discard them. Git GC cannot bound retained promisor objects, so expiry deliberately starts fresh.This PR's first commit refactors out the existing git repo management behaviour into a struct called gitRepoCache. Then the following commits add the behaviour around keeping the clone around.
Part of CU-3324