Skip to content

Consume flat timestamp-ordered histories during replay - #882

Draft
frankmcsherry wants to merge 1 commit into
TimelyDataflow:master-nextfrom
frankmcsherry:codex/ddir-consuming-history
Draft

frankmcsherry wants to merge 1 commit into
TimelyDataflow:master-nextfrom
frankmcsherry:codex/ddir-consuming-history

Conversation

@frankmcsherry

Copy link
Copy Markdown
Member

History replay retains original edits plus a second timestamp/index representation, then clones timestamps/differences into the replay buffer. Store edits in descending timestamp order with prefix meets, and consume each edit into the buffer as it is replayed. This replaces the representation with +21/-34 lines in one file.

The internal replay operation now consumes edits; it must be preceded by reloading the history. Current cursor reduce and join callers clear/reload before replay; the join small-cross-product path reads edits before replay. This touches shared DD history machinery and is intentionally a separate draft for discussion.

Validation: DD's full default suite, cargo test --locked --offline -p differential-dataflow, passed on this independent branch: 87 regular tests and 54 doc tests, including cursor reduce/join, their oracles, and SCC. Three regular tests and six doc examples remain ignored by the suite. Timely revision: 4eb3ee09681c209ebcaeced73548a9fbf8e1c71e.

Earlier isolated measurements atop #880 plus grouped pending times showed Corgi SCC startup 5.496 → 5.420 seconds (-1.4%) and ten updates 5.714 → 5.555 seconds (-2.8%); compiled SCC was essentially flat. These are three alternating pairs on Apple M4, one worker, release/LTO, 500k nodes/1M edges, ten rounds of 1,000 replacements. They are not measurements of a combined ownership/direct-evaluation bundle, and should not be added to that bundle's gains. Other cursor-heavy workloads remain useful follow-up validation.

Based directly on master-next at dc3733d (#880 merged); no dependency on the pending or direct-evaluation PRs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant