[enhancement](scan) Backport Parquet V2 direct predicate filtering - #66360
[enhancement](scan) Backport Parquet V2 direct predicate filtering#66360Gabriel39 wants to merge 4 commits into
Conversation
Keep identity selections implicit, refresh late predicates at row-group boundaries, and reuse dictionary predicate bitmaps within a query.
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
|
/review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Request changes.
I found two PR-introduced issues in the late runtime-filter refresh path:
- [P1] A COUNT(*) carrier that is still represented by synthesized placeholder defaults can be promoted to a late predicate while the active Parquet Row Group continues returning defaults; Scanner immediately evaluates the new RF and can silently drop or retain the wrong rows.
- [P2] Activating the new request retains footer-planning and adaptive-prefetch state derived from the old predicate snapshot, losing cheap pruning and potentially warming wide lazy output before a selective RF is sampled.
Critical checkpoint summary:
- Runtime-filter lifecycle and COUNT(*) semantics: blocked by the placeholder transition above.
- Parquet Row Group pruning and adaptive lazy-read state: needs request versioning or equivalent invalidation at activation.
- Direct OR/NULL behavior, fused SelectionVector compaction, dictionary-reader reuse, and MergeRange ordering: no additional PR-introduced defect after focused recheck.
- Mapping/layout compatibility, condition-cache handling, JNI/Hudi/Paimon forwarding, and expression ownership: no additional issue found.
- Coverage: the changed tests exercise request queuing/activation, but not mid-Row-Group COUNT(*) result correctness or request-versioned multi-Row-Group I/O.
- User focus: no additional focus was supplied; the full 39-file diff was reviewed.
- Validation: static review only as required; no builds or tests were run.
| } | ||
| RETURN_IF_ERROR(customize_file_scan_request(refreshed_request.get())); | ||
| if (_file_scan_request == nullptr || | ||
| !same_physical_scan_layout(*refreshed_request, *_file_scan_request)) { |
There was a problem hiding this comment.
[P1] Keep real COUNT() carrier values available until the refreshed request activates. When an RF promotes the current placeholder slot to a predicate, this compatibility check still accepts the request because positions/projections match. The Parquet scheduler does not activate it until the next row-group boundary, while Scanner::_filter_output_block() starts evaluating the new RF immediately; the active request has no reader for this slot and materializes defaults. For example, a current group containing ids [1,10] with a late id > 5 can test [0,0] and discard the matching row. Please avoid placeholdering possible RF carriers while filters are pending (or otherwise make the transition/fallback expose real values), and add a mid-row-group COUNT() refresh result test.
| } | ||
| // Column readers and predicate schedules retain request-derived state for one row group. Swap | ||
| // only after they are gone; the refreshed request may promote a lazy column to a predicate. | ||
| _active_request = std::move(_pending_request); |
There was a problem hiding this comment.
[P2] Invalidate all predicate-snapshot-derived optimization state when this request becomes active. The unopened Row Group plans were footer-pruned with the old request, but their deferred finalization is EXPENSIVE_ONLY, so a late RF whose min/max rejects a group never gets the cheap footer-statistics pass. The old _predicate_survival_ratio/position stats also survive, allowing pre-RF dense batches to trigger full lazy-output prefetch before the new selective RF is sampled. Please make remaining planning/adaptive state request-versioned (or rerun current-request footer pruning and reset/preserve stats only when predicates are proven equivalent), with a multi-group late-RF I/O regression.
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
db8ebab to
4b5c6c4
Compare
2 participants
Proposed changes
Backport four selected commits from #66261 to
branch-4.1in their original order.Test
./run-be-ut.sh --run '--filter=FileScannerV2Test.*:*Parquet*:*TableReaderTest.*:Hudi*ReaderTest.*:Paimon*ReaderTest.*:SelectionVectorTest.*:DictionaryFilterCostTest.*'git diff origin/branch-4.1...HEAD --check