Skip to content

[Python] Fix execution boundary regressions and restore Native CI - #10782

Merged
JingsongLi merged 2 commits into
apache:masterfrom
JingsongLi:codex/pypaimon-master-ci-fixes-20261023
Oct 24, 2026
Merged

JingsongLi merged 2 commits into
apache:masterfrom
JingsongLi:codex/pypaimon-master-ci-fixes-20261023

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Restore Python and Native regression coverage after separating public APIs from execution backends:

  • Make deferred vector and full-text factories pickleable while preserving captured query configuration and lazy executor construction.
  • Preserve the original commit error when the optional Rust conflict exception is unavailable.
  • Use the vector SDK batch ABI for a single query too, preserving short results when null vectors or deletion vectors leave fewer than K matches.
  • Check custom query authorization before raw/all-file Native planning and preserve cached restrictions when access is denied.

Align backend guards, public updater wrapper checks and lazy-import mocks with the current execution classes, retaining data, cleanup, file-layout and no-fallback assertions. Correct the stale-bucket injection target and late write-type error expectation. Set DAFT_RUNNER=native in Native CI to avoid retaining an inferred Ray runner across test cluster shutdowns; explicit Ray-runner tests remain covered.

For the malformed Lance descriptor fixture, use only the Python writer to seed invalid legacy bytes. Native planning/reading, export rejection and create/overwrite atomicity assertions remain enabled and independently verified.

Dependencies: Native CI builds paimon-rust main and needs apache/paimon-rust#1211 for three indexed partition-filter cases. Rust's advisory gate independently needs apache/paimon-rust#1212. Final local Native validation uses the wheel built from vector-fix commit 188a754573f90d113a560345edea126e999cb5f1.

Tests

  • Full Python 3.10 suite without Rust: 10,635 passed, 705 skipped, 11,755 native cases deselected, exit status 0.
  • Full Native suite with all five execution flags, the Rust vector fix and the CI runner setting: 23,016 passed, 89 skipped, exit status 0. Native execution counters: 23,508 plans; 19,617 reads; 20,054 writes; 3,621 commits; all seven update kinds exercised.
  • Lance 13.0.0 with Native plan/read/write enabled: 120 passed after reproducing and fixing the two malformed-fixture setup failures. Independent verification confirmed both cases reach export validation using Native plan/read and retain destination atomicity.
  • LeRobot 0.6.0 and 0.6.1: 145 passed each. PyArrow 22 nested MAP field names: 12 passed, 2 capability skips. Torch: 71 passed. Temporal/LeRobot targeted regression: 40 passed; non-UTC temporal casts and golden timestamp serialization passed. Packaging: 9 passed.
  • Community CI on the initial implementation commit passed Python 3.6.15/3.7/3.10/3.11/3.12/3.13, all four PyArrow compatibility versions, all three LeRobot versions, Torch, packaging and Ray compatibility. Its Native job exposed the three Rust dependency failures and the two Lance fixture failures addressed above; the fixture-only follow-up has been pushed.
  • Java prerequisites passed with default checks and JDK 8. Java/Python interoperability: 30 groups passed; 2 Lumina groups cannot load native libraries on this macOS ARM host. Lumina Python wheels are also unavailable for this host.
  • License headers, Flake8, git diff --check and independent review passed.

Local full-suite runs used macOS ARM, Python 3.10 without Rust and Python 3.13 with Rust. Lance/LeRobot supplemental runs used PyArrow 22. Skips include unavailable cloud credentials/native libraries and version-specific checks; Lance and timezone cases were exercised separately.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi merged commit 40b011d into apache:master Oct 24, 2026
18 of 20 checks passed
QuakeWang added a commit to QuakeWang/paimon that referenced this pull request Oct 24, 2026
Keep the reader and Native test identical to upstream after apache#10782.
Retain real-SDK sparse-filter and Python deletion-vector regression coverage.

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants