Move reduce timestamps and directly evaluate single dominating seeds - #883
Merged
frankmcsherry merged 4 commits intoSep 15, 2026
Merged
frankmcsherry merged 4 commits into
frankmcsherry merged 4 commits into
Conversation
frankmcsherry
force-pushed
the
codex/ddir-reduce-direct
branch
from
September 15, 2026 12:59
8211936 to
fc2acae
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A key with one seed that dominates every selected input/output timestamp has exactly one possible evaluation. Reconcile its input/output ranges directly, or retain the seed if it has not crossed the frontier, while leaving the general partial-order walk for all other keys. This avoids building and replaying histories for the single-evaluation case.
Also move consumed timestamps through two ownership boundaries: selected rows into Corgi's presentation merge, and bridge rows into histories. Neither consumed source needs its old timestamp again. The production diff is +60/-35 (+25 net) across int_proxy reduce and Corgi presentation.
Three focused tests (63 lines) exercise four conditions: a seed beyond upper must remain pending; that seed subsequently becomes due with no novel input; input and compacted prior-output records exactly at the seed must participate; and an incomparable record requires the general walk, including a correction at the joined time. Existing partial-order/window-size, cancellation, collision and backend tests remain in place.
Validation:
cargo test --locked --offline -p differential-dataflow -p interactivepassed (196 regular tests and 54 doc tests; 8 regular and 6 doc tests remain ignored). The incomparable-record test uses a two-element compaction frontier. Validation was rerun after rebasing onto merged #881, including the grouped-pending tests and all three direct-evaluation tests.Original isolated SCC measurement before the rebase, against #880, with neither grouped pending nor the history replacement on either side. These are not measurements of the incremental gain above #881:
Apple M4, one worker, release/LTO, 500k nodes/1M edges, ten rounds of 1,000 replacements, three alternating process pairs, same harness/compiler/Timely 4eb3ee09681c209ebcaeced73548a9fbf8e1c71e. Output checksums match and there were no pageout/swap counter increases. These are measured combined effects of this PR, not sums of the individual experiments. The baseline executable was built from #880's reviewed head; its repository contents match merged dc3733d.
Based on
master-nextat 680500b (#881 merged). Direct evaluation reads due times through the grouped pending storage now in the base. The shared-history representation change in #882 is excluded.