perf(storage): parallelize Claude Code transcript listing - #5096
Open
E2ern1ty wants to merge 1 commit into
Open
Conversation
Listing walked the projects directory and every transcript one await at a time, so a full scan serialized its round trips through the thread pool instead of overlapping them — and the catalog is listed once per search term, so the walk is paid again on files that did not change. With 1000 local transcripts the warm relist drops from ~20ms to ~5ms here, and the first listing from ~155ms to ~120ms. The directory walk, the per-file stats, and the summary derivations are now concurrent. Behavior is unchanged: per-item failures are still swallowed individually, the newest-wins dedup is order-independent (its tie-break names the path, not the iteration order), and the final sort is stable over the same input order. #parse also reuses the size the cache check already stat'ed rather than stat'ing the same file twice. Generated-by: ZCode
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Listing Claude Code transcripts walked the projects directory and every transcript one
awaitat a time, so a full scan serialized its round trips through the thread pool instead of overlapping them — and as the adapter's own comments note, the catalog is listed once per search term, so the walk is paid again on files that did not change.The directory walk, the per-file stats, and the summary derivations are now concurrent:
listSessionsderives all summaries viaPromise.allinstead of oneawaitper file.#summaryOfnever throws and only touches the summary cache at distinct keys (one file per session id), so the calls are independent; results keep the original order and the final sort is stable.#transcriptFileslists project directories and stats candidate files concurrently. The newest-wins dedup stays a sequential, order-independent reduce (its tie-break names the path, not the iteration order), so completion order cannot change which file wins.#parsetakes the file size the cache check already stat'ed, instead of stat'ing the same file a second time.Measured on 1000 seeded transcripts (20 projects, ~50 records each): warm relist (all files cached) drops from ~20 ms to ~5 ms; the first listing from ~155 ms to ~120 ms.
Also adds a test pinning the previously-untested dedup rule that the concurrent walk must preserve: one session id under two project directories must resolve to the newest copy in both list and read.
Verification
npm --workspace @maka/storage run typecheck— cleannpm --workspace @maka/storage run test:dist— 1215 pass / 0 fail, including the new testnpx biome checkon both changed files — cleanlistSessions20–22 ms → 5–8 ms; cold 155–156 ms → 118–128 msAI use
Select exactly one:
Tool(s) and scope: ZCode authored the change, the new test, and this description; benchmark numbers were measured locally on real runs.
Checklist
Note: this is a behavior-preserving performance change, so no test fails without it; the new test pins the newest-wins dedup invariant the concurrent walk must uphold.
Does this PR entail a change in behavior?