Skip to content

Record why source checkpoints need bounded continuation - #1213

Draft
Floating-Y wants to merge 1 commit into
deeplethe:devfrom
Floating-Y:fix/source-window-1212
Draft

Floating-Y wants to merge 1 commit into
deeplethe:devfrom
Floating-Y:fix/source-window-1212

Conversation

@Floating-Y

Copy link
Copy Markdown
Contributor

Why

Sources currently import only the first 200 records yet advance the successful checkpoint over the unprocessed tail. Retaining the previous lower bound also cannot get past the readers' fixed ten-page caps.

Refs #1212. This ADR-only draft does not close the issue or implement its fix. The model proposal is under discussion.

What changes

Record ADR 0067 as Proposed and index it under sources. It explains bounded private continuation state, fixed window bounds, first-batch checkpointing, crash replay through existing document/version/job deduplication, pagination drift, and one bounded recovery scan for historical omissions.

The private field and incomplete-batch status remain discussion items. No migration, dependency, public API, production reader, or checkpoint implementation changes are included. A production implementation will follow once this ADR has landed.

How it was checked

Baseline: upstream dev 259931ac702623351c2ed4e6487474790e57a6b0. Windows; Rust/Cargo 1.98.1; dedicated PostgreSQL 16.15 with pgvector 0.8.6 and UTOPIA_TEST_REQUIRE_DB=1.

  • Fresh database migrations passed.
  • In the separate working tree, the original three checkpoint tests passed. Two new, uncommitted custom 201/450 regressions failed as expected when the current cursor advanced after only 200 records. These failing tests are retained for the implementation and are not part of this documentation PR.
  • cargo fmt --all --check, git diff --check, and cargo clippy --locked -p utopia-server -p utopia-store --all-targets -- -D warnings passed.
  • No workspace-wide or live-provider test result is claimed.

Before review

  • Every commit is signed off (b85855c).
  • Base is dev.
  • The ADR status is Proposed and does not claim an accepted or implemented contract.
  • Resolve private continuation storage and incomplete-batch status in the issue discussion.
  • Settle and land the ADR before implementation.

Signed-off-by: Floating-Y <118035379+Floating-Y@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant