Skip to content

Fix partition field mapping for vector search source plans - #1211

Merged
JingsongLi merged 4 commits into
apache:mainfrom
JingsongLi:codex/search-partition-predicate-20261024
Oct 24, 2026
Merged

JingsongLi merged 4 commits into
apache:mainfrom
JingsongLi:codex/search-partition-predicate-20261024

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Indexed vector search can lose matching rows when a partition column's position in the table schema differs from its position in the partition row. For example, filtering pt in a schema id, embedding, pt can return no results when an exact scalar index filter selects a source plan.

Brief change log

Remap the partition predicate from partition-row positions back to complete table-schema positions before collecting allowed source row IDs.

Add regression coverage for single and batch vector searches in fast, full and detail modes, checking both scores and materialized rows.

Tests

  • The 6 new regression cases fail against main 96e06d764365b59c9895a45692819efccf209b23 and pass with this change.
  • cargo test -p paimon --lib table::de_vector_read::tests: 56 passed.
  • Python binding test_vector_search.py, using a wheel built from this commit: 15 passed.
  • cargo fmt --all --check and git diff --check: passed.
  • Combined with [Python] Fix execution boundary regressions and restore Native CI paimon#10782, the full PyPaimon Native suite passed 23,016 tests with 89 skips.

The branch includes the dependency security updates and regenerated inventories from #1212 so CI can validate this vector fix together with its prerequisites. The functional vector patch remains byte-for-byte identical to commit 188a754573f90d113a560345edea126e999cb5f1. Merge #1212 first.

After combining the changes, the CI-pinned Rust 1.98.0 workspace Clippy check passed after rebuilding paimon, dependency-report verification passed, and all 56 vector unit tests passed again. The original vector-only CI run passed all configured build, unit and integration jobs; the combined branch has triggered a new CI run.

API and Format

No public API or storage-format changes.

Documentation

Bug fix; no documentation changes required.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi merged commit 9271c51 into apache:main Oct 24, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants