Skip to content

[python] Dispatch captured row ID and bucket execution - #10778

Merged
JingsongLi merged 1 commit into
apache:masterfrom
JingsongLi:codex/native-row-id-routing
Oct 23, 2026
Merged

JingsongLi merged 1 commit into
apache:masterfrom
JingsongLi:codex/native-row-id-routing

Conversation

@JingsongLi

Copy link
Copy Markdown
Contributor

Purpose

Move the remaining distributed row-ID range lookup and deletion grouping behind execution dispatch, and finish placing the corresponding Python bucket execution implementations under execution/python. Public PyPaimon and Ray API signatures remain unchanged.

Changes

  • Give captured row-ID plans matching Python and Native route, route_by_bucket and read-only select execution contracts.
  • Delegate Native routing to Core's existing RowIdPlan with Arrow inputs. Worker serialization drops only the SDK handle and reconstructs the same captured typed plan; it does not replan, reselect a backend or replay execution errors in Python.
  • Remove repeated NumPy range lookup from Ray read/update/delete callbacks and replace pickle/base64 deletion keys with typed per-partition/bucket routing IDs. Keep Ray scheduling and existing worker authorization, liveness checks and commit messages.
  • Keep complete file groups, deletion vectors, authorization, paths and snapshot metadata during read selection.
  • Match Core's normal-anchor ownership when a Blob/vector file spans multiple groups: attach it to overlapping groups without joining anchors or filling row-ID gaps. Preserve planning work and LIMIT statistics while copying and refreshing the selected plan's final output counters, including after pickle.
  • Move Python-only bucket hashing, HASH index restoration and stateful assignment into execution/python. Keep shared limits, validation and read-view configuration outside execution. Update internal imports and packaging without compatibility reexports or SDK version probes.

Verification

  • 5 real Ray cases passed: cross-file reads and updates, updates with deletion vectors, partial-column evolution files and Blob reads.
  • 295 current-SDK integration tests passed, exercising routing, plan serialization, captured-snapshot updates/deletes, selected reads and existing distributed bucket execution. No Native extension was installed or rebuilt for these Python tests.
  • Independent Python row-ID regression collection: 87 passed, 4 skipped without an installed SDK; those Native paths passed in the joint collection. Bucket regressions: 158 passed (collections overlap).
  • Actual wheel-content validation passed. Changed-file Flake8 and diff checks passed.

Depends on apache/paimon-rust#1209. This PR will remain draft until the companion Rust API reaches main. Native CI continues to build Rust main; no CI pin or compatibility fallback is added.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi merged commit d706289 into apache:master Oct 23, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants