Is your feature request related to a problem or challenge?
Ordinary joins on multiple equality columns can miss opportunities to skip data. For example:
SELECT *
FROM fact f
JOIN dimension d
ON f.a = d.x
AND f.b = d.y;
When the build-side input is small enough, DataFusion collects its key combinations and creates a filter like:
(f.a, f.b) IN ((1, 10), (2, 20))
This correctly filters rows, but Bloom-filter pruning cannot extract the allowed values for each column. We can therefore read row groups that cannot contain a
Describe the solution you'd like
We can extract the allowed values for each column and use them
a IN (1, 2)
b IN (10, 20)
while also keeping the original filter so that we don't have any false positive matches
Describe alternatives you've considered
Adding separate row filters for each column. This would repeat checks when the original tuple filter already checks the complete combination.
Additional context
No response
Is your feature request related to a problem or challenge?
Ordinary joins on multiple equality columns can miss opportunities to skip data. For example:
When the build-side input is small enough, DataFusion collects its key combinations and creates a filter like:
This correctly filters rows, but Bloom-filter pruning cannot extract the allowed values for each column. We can therefore read row groups that cannot contain a
Describe the solution you'd like
We can extract the allowed values for each column and use them
while also keeping the original filter so that we don't have any false positive matches
Describe alternatives you've considered
Adding separate row filters for each column. This would repeat checks when the original tuple filter already checks the complete combination.
Additional context
No response