Skip to content

Move reduce timestamps and directly evaluate single dominating seeds - #883

Merged
frankmcsherry merged 4 commits into
TimelyDataflow:master-nextfrom
frankmcsherry:codex/ddir-reduce-direct
Sep 15, 2026
Merged

frankmcsherry merged 4 commits into
TimelyDataflow:master-nextfrom
frankmcsherry:codex/ddir-reduce-direct

Conversation

@frankmcsherry

@frankmcsherry frankmcsherry commented Sep 15, 2026

Copy link
Copy Markdown
Member

A key with one seed that dominates every selected input/output timestamp has exactly one possible evaluation. Reconcile its input/output ranges directly, or retain the seed if it has not crossed the frontier, while leaving the general partial-order walk for all other keys. This avoids building and replaying histories for the single-evaluation case.

Also move consumed timestamps through two ownership boundaries: selected rows into Corgi's presentation merge, and bridge rows into histories. Neither consumed source needs its old timestamp again. The production diff is +60/-35 (+25 net) across int_proxy reduce and Corgi presentation.

Three focused tests (63 lines) exercise four conditions: a seed beyond upper must remain pending; that seed subsequently becomes due with no novel input; input and compacted prior-output records exactly at the seed must participate; and an incomparable record requires the general walk, including a correction at the joined time. Existing partial-order/window-size, cancellation, collision and backend tests remain in place.

Validation: cargo test --locked --offline -p differential-dataflow -p interactive passed (196 regular tests and 54 doc tests; 8 regular and 6 doc tests remain ignored). The incomparable-record test uses a two-element compaction frontier. Validation was rerun after rebasing onto merged #881, including the grouped-pending tests and all three direct-evaluation tests.

Original isolated SCC measurement before the rebase, against #880, with neither grouped pending nor the history replacement on either side. These are not measurements of the incremental gain above #881:

#880 baseline Ownership moves + direct evaluation Reduction
Startup 5.696 s 4.978 s 12.6%
Ten updates 7.039 s 6.458 s 8.3%
Peak RSS 1464 MiB 1454 MiB 0.7%

Apple M4, one worker, release/LTO, 500k nodes/1M edges, ten rounds of 1,000 replacements, three alternating process pairs, same harness/compiler/Timely 4eb3ee09681c209ebcaeced73548a9fbf8e1c71e. Output checksums match and there were no pageout/swap counter increases. These are measured combined effects of this PR, not sums of the individual experiments. The baseline executable was built from #880's reviewed head; its repository contents match merged dc3733d.

Based on master-next at 680500b (#881 merged). Direct evaluation reads due times through the grouped pending storage now in the base. The shared-history representation change in #882 is excluded.

@frankmcsherry
frankmcsherry force-pushed the codex/ddir-reduce-direct branch from 8211936 to fc2acae Compare September 15, 2026 12:59
@frankmcsherry
frankmcsherry merged commit 51c4dd0 into TimelyDataflow:master-next Sep 15, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant