Feat: add dictionaries as a supported group column type by Rich-T-kid · Pull Request #23187 · apache/datafusion

Rich-T-kid · 2026-06-25T13:52:52Z

Which issue does this PR close?

works towards closing feat: add GroupColumn support for List / Struct / Map / FixedSizeList in multi-column GROUP BY #22682.
replacement for Optimize Dictionary groupings #21765
- Optimize Dictionary groupings #21765 (review)

Rationale for this change

This PR introduces a specialized GroupColumn implementation for dictionary-typed columns inside GroupValuesColumn, allowing dictionary columns to participate in the columnar, vectorized aggregation path instead of the row-based fallback.

The Implementation is only about 175+ lines of code. the remaining LOC is adding extensive test at the GroupColumn trait level as well as testing the GroupValuesColumn GroupValues trait and how it inter-opts with multi-dictionary group by's.

What changes are included in this PR?

Adds a DictionaryGroupValueBuilder struct implementing the GroupColumn trait for Dictionary-typed group-by columns, supporting a configurable subset of value types
Extends the type-check gate in GroupValuesColumn::try_new (the matches! block) to accept Dictionary(_, value_type) where value_type is already supported.
Adds schema-level support so emitted dictionary group key columns round-trip through the output schema correctly
removes casting thats done for each dictionary array in emit

Are these changes tested?

yes. a majority of this PR is test

Are there any user-facing changes?

no. this is a pure perf boost for users.

…ore inner append

Rich-T-kid · 2026-07-01T04:28:17Z

@kumarUjjawal could you run the dictionary benchmarks on this PR? Thx

Rich-T-kid · 2026-07-01T05:08:03Z

            }
        }
+        DataType::Dictionary(key_dt, value_dt) => {
+            let new_field = Field::new("", *value_dt.clone(), true);


Since this field is never read again it may be fine to ignore the name field.

should be weary of similar issues to #21765 (comment)

Rich-T-kid · 2026-07-01T13:33:07Z

@kumarUjjawal wanted to bump this 😄

geoffreyclaude · 2026-07-01T15:06:25Z

run benchmark dictionary_group_values

adriangbot · 2026-07-01T15:09:30Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4856707692-778-smmmp 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing rich-T-kid/dictionary-groupValuesColumn-impl (eb41915) to 01bf68c (merge-base) diff using: dictionary_group_values
Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-07-01T15:21:31Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                                                             HEAD                                   rich-T-kid_dictionary-groupValuesColumn-impl
-----                                                                             ----                                   --------------------------------------------
dict_intern_emit/intern_emit/size_65536_card_1000_null_0.00                       1.15    869.8±5.08µs 71.9 MElem/sec    1.00  756.8±139.44µs 82.6 MElem/sec
dict_intern_emit/intern_emit/size_65536_card_20_null_0.00                         1.22    779.9±5.80µs 80.1 MElem/sec    1.00   636.7±43.49µs 98.2 MElem/sec
dict_intern_emit/intern_emit/size_65536_card_300_null_0.00                        1.17    809.2±5.84µs 77.2 MElem/sec    1.00  692.8±108.32µs 90.2 MElem/sec
dict_intern_emit/intern_emit/size_65536_card_65536_null_0.00                      2.45      6.3±0.01ms 10.0 MElem/sec    1.00      2.6±0.04ms 24.5 MElem/sec
dict_intern_emit/intern_emit/size_65536_card_75_null_0.00                         1.24    800.2±6.35µs 78.1 MElem/sec    1.00    645.5±6.09µs 96.8 MElem/sec
dict_intern_emit/intern_emit/size_8192_card_1000_null_0.00                        1.42    160.1±0.87µs 48.8 MElem/sec    1.00   112.7±15.66µs 69.3 MElem/sec
dict_intern_emit/intern_emit/size_8192_card_20_null_0.00                          1.28    104.0±1.39µs 75.1 MElem/sec    1.00     81.5±0.45µs 95.8 MElem/sec
dict_intern_emit/intern_emit/size_8192_card_300_null_0.00                         1.30    121.2±1.05µs 64.5 MElem/sec    1.00     92.9±9.63µs 84.1 MElem/sec
dict_intern_emit/intern_emit/size_8192_card_75_null_0.00                          1.17    109.7±1.26µs 71.2 MElem/sec    1.00    94.1±13.82µs 83.0 MElem/sec
dict_intern_emit/intern_emit/size_8192_card_8192_null_0.00                        2.66    678.0±2.09µs 11.5 MElem/sec    1.00   255.3±12.37µs 30.6 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_65536_card_1000_null_0.10     1.04      4.4±0.02ms 57.4 MElem/sec    1.00      4.2±0.01ms 59.7 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_65536_card_20_null_0.10       1.02      4.1±0.02ms 60.5 MElem/sec    1.00      4.1±0.01ms 61.5 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_65536_card_300_null_0.10      1.01      4.2±0.02ms 59.2 MElem/sec    1.00      4.2±0.01ms 60.1 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_65536_card_65536_null_0.10    1.54     16.5±0.05ms 15.2 MElem/sec    1.00     10.7±0.06ms 23.4 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_65536_card_75_null_0.10       1.01      4.2±0.02ms 59.3 MElem/sec    1.00      4.2±0.01ms 60.2 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_8192_card_1000_null_0.10      1.13    618.7±3.27µs 50.5 MElem/sec    1.00    549.9±6.40µs 56.8 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_8192_card_20_null_0.10        1.00    512.1±3.06µs 61.0 MElem/sec    1.00    511.2±3.10µs 61.1 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_8192_card_300_null_0.10       1.05    550.0±3.63µs 56.8 MElem/sec    1.00    522.5±2.60µs 59.8 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_8192_card_75_null_0.10        1.02    526.6±2.85µs 59.3 MElem/sec    1.00    515.0±2.45µs 60.7 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_8192_card_8192_null_0.10      1.74   1479.0±6.20µs 21.1 MElem/sec    1.00    847.9±5.26µs 36.9 MElem/sec

Resource Usage

dictionary_group_values — base (merge-base)

Metric	Value
Wall time	380.1s
Peak memory	592.6 MiB
Avg memory	94.8 MiB
CPU user	227.9s
CPU sys	14.8s
Peak spill	0 B

dictionary_group_values — branch

Metric	Value
Wall time	330.1s
Peak memory	464.6 MiB
Avg memory	67.9 MiB
CPU user	225.8s
CPU sys	19.2s
Peak spill	0 B

File an issue against this benchmark runner

Rich-T-kid · 2026-07-01T18:55:54Z

@codex review

kumarUjjawal · 2026-07-02T03:17:29Z

@kumarUjjawal wanted to bump this 😄

@Rich-T-kid Thank you! I have been sick so I won't be available for review. I will probably get back next week.

Rich-T-kid · 2026-07-06T20:06:26Z

Latest PR speeds up the low cardinality case

Rich-T-kid · 2026-07-06T20:07:27Z

@geoffreyclaude could you run the benchmarks command again when you get a chance. Thanks 🚀

geoffreyclaude · 2026-07-06T20:12:42Z

run benchmark dictionary_group_values

adriangbot · 2026-07-06T20:15:36Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4897199089-877-zms2m 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing rich-T-kid/dictionary-groupValuesColumn-impl (3d1e1c9) to 01bf68c (merge-base) diff using: dictionary_group_values
Results will be posted here when complete

File an issue against this benchmark runner

Rich-T-kid

Revision #4 Last revision before making this open for review

Rich-T-kid · 2026-07-06T20:16:14Z

+    cached_combined: Option<ArrayRef>,
+    /// Per-call group equality cache for the low-cardinality path. `Mutex`
+    /// because `vectorized_equal_to` takes `&self`; always uncontended.
+    group_eq_cache: Mutex<Vec<Option<bool>>>,


This is a work around needed because vectorized_equal_to takes in a &self as opposed to &mut self. This means the alternative is needing to allocate target_batch_size arrays for each intern() call instead of re-using the vector.

Originally planned to use RefCell but its not Send + Sync

this shouldn't incur any overhead since there no contention but I'm happy to explore other ideas.

adriangbot · 2026-07-06T20:25:58Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                                                             HEAD                                   rich-T-kid_dictionary-groupValuesColumn-impl
-----                                                                             ----                                   --------------------------------------------
dict_intern_emit/intern_emit/size_65536_card_1000_null_0.00                       1.23    871.4±8.87µs 71.7 MElem/sec    1.00    707.5±5.34µs 88.3 MElem/sec
dict_intern_emit/intern_emit/size_65536_card_20_null_0.00                         1.30    790.0±9.68µs 79.1 MElem/sec    1.00    608.0±7.50µs 102.8 MElem/sec
dict_intern_emit/intern_emit/size_65536_card_300_null_0.00                        1.23   815.1±10.24µs 76.7 MElem/sec    1.00    665.2±5.90µs 94.0 MElem/sec
dict_intern_emit/intern_emit/size_65536_card_65536_null_0.00                      2.51      6.2±0.02ms 10.0 MElem/sec    1.00      2.5±0.01ms 25.1 MElem/sec
dict_intern_emit/intern_emit/size_65536_card_75_null_0.00                         1.23    804.8±8.79µs 77.7 MElem/sec    1.00    655.6±9.93µs 95.3 MElem/sec
dict_intern_emit/intern_emit/size_8192_card_1000_null_0.00                        1.35    161.5±1.39µs 48.4 MElem/sec    1.00   119.8±17.59µs 65.2 MElem/sec
dict_intern_emit/intern_emit/size_8192_card_20_null_0.00                          1.31    104.8±1.68µs 74.5 MElem/sec    1.00     80.0±0.85µs 97.7 MElem/sec
dict_intern_emit/intern_emit/size_8192_card_300_null_0.00                         1.34    123.0±1.59µs 63.5 MElem/sec    1.00    92.0±10.05µs 84.9 MElem/sec
dict_intern_emit/intern_emit/size_8192_card_75_null_0.00                          1.29    112.0±6.33µs 69.7 MElem/sec    1.00    86.6±30.70µs 90.2 MElem/sec
dict_intern_emit/intern_emit/size_8192_card_8192_null_0.00                        2.70    681.0±3.01µs 11.5 MElem/sec    1.00    252.0±1.80µs 31.0 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_65536_card_1000_null_0.10     1.45      4.4±0.03ms 57.1 MElem/sec    1.00      3.0±0.01ms 82.8 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_65536_card_20_null_0.10       1.58      4.2±0.02ms 60.1 MElem/sec    1.00      2.6±0.02ms 94.8 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_65536_card_300_null_0.10      1.48      4.2±0.02ms 58.9 MElem/sec    1.00      2.9±0.01ms 87.4 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_65536_card_65536_null_0.10    1.40     17.0±0.13ms 14.7 MElem/sec    1.00     12.2±0.13ms 20.5 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_65536_card_75_null_0.10       1.51      4.2±0.03ms 59.1 MElem/sec    1.00      2.8±0.02ms 89.1 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_8192_card_1000_null_0.10      1.29    620.6±3.17µs 50.4 MElem/sec    1.00    480.4±1.96µs 65.1 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_8192_card_20_null_0.10        1.54    513.2±3.32µs 60.9 MElem/sec    1.00    332.5±5.61µs 94.0 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_8192_card_300_null_0.10       1.48    549.7±3.56µs 56.8 MElem/sec    1.00    370.5±1.52µs 84.3 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_8192_card_75_null_0.10        1.53    528.9±3.45µs 59.1 MElem/sec    1.00    344.6±4.98µs 90.7 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_8192_card_8192_null_0.10      1.52   1486.9±5.41µs 21.0 MElem/sec    1.00    975.5±4.94µs 32.0 MElem/sec

Resource Usage

dictionary_group_values — base (merge-base)

Metric	Value
Wall time	290.1s
Peak memory	587.7 MiB
Avg memory	124.8 MiB
CPU user	227.7s
CPU sys	15.5s
Peak spill	0 B

dictionary_group_values — branch

Metric	Value
Wall time	320.1s
Peak memory	486.4 MiB
Avg memory	71.3 MiB
CPU user	210.2s
CPU sys	15.2s
Peak spill	0 B

File an issue against this benchmark runner

Rich-T-kid · 2026-07-06T20:50:20Z

🤔 benchmarks show good improvement in every case but it should be much larger. Currently every intern() call causes the cache to be rebuilt from the bottom up. Instead of a flat vector it may make sense to use a hashtable that last in-between calls.

Rich-T-kid · 2026-07-07T17:20:10Z

more test that cover the changes of this PR are #23280

Rich-T-kid · 2026-07-07T17:34:04Z

+pub struct DictionaryGroupValuesColumn<K: ArrowDictionaryKeyType + Send + Sync> {
+    inner: Box<dyn GroupColumn>,
+    null_array: ArrayRef,
+    // Mutex is required because `vectorized_equal_to` takes `&self` (GroupColumn trait


RefCell doesn't work here because its not sync

std::sync::RwLock may also be an option

Rich-T-kid · 2026-07-07T17:39:01Z

PR is ready for review 🚀
@geoffreyclaude could you run the benchmarks one more time to verify theres no regressions 🙇

geoffreyclaude · 2026-07-07T17:40:05Z

PR is ready for review 🚀
@geoffreyclaude could you run the benchmarks one more time to verify theres no regressions 🙇

Did you run them yourself locally? What are your results?

Rich-T-kid · 2026-07-07T18:11:32Z

PR is ready for review 🚀
@geoffreyclaude could you run the benchmarks one more time to verify theres no regressions 🙇

Did you run them yourself locally? What are your results?

@geoffreyclaude yes. These are the results compared to main

geoffreyclaude · 2026-07-07T18:17:34Z

run benchmark dictionary_group_values

adriangbot · 2026-07-07T18:20:37Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4907084674-898-7668q 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing rich-T-kid/dictionary-groupValuesColumn-impl (894a0a6) to 01bf68c (merge-base) diff using: dictionary_group_values
Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-07-07T18:31:24Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                                                             HEAD                                   rich-T-kid_dictionary-groupValuesColumn-impl
-----                                                                             ----                                   --------------------------------------------
dict_intern_emit/intern_emit/size_65536_card_1000_null_0.00                       1.00    869.0±4.52µs 71.9 MElem/sec    1.18   1025.1±8.39µs 61.0 MElem/sec
dict_intern_emit/intern_emit/size_65536_card_20_null_0.00                         1.00    781.1±7.66µs 80.0 MElem/sec    1.14    891.0±6.32µs 70.1 MElem/sec
dict_intern_emit/intern_emit/size_65536_card_300_null_0.00                        1.00    815.4±7.88µs 76.7 MElem/sec    1.16    945.3±6.21µs 66.1 MElem/sec
dict_intern_emit/intern_emit/size_65536_card_65536_null_0.00                      2.49      6.4±0.07ms  9.8 MElem/sec    1.00      2.6±0.09ms 24.4 MElem/sec
dict_intern_emit/intern_emit/size_65536_card_75_null_0.00                         1.00    798.9±8.58µs 78.2 MElem/sec    1.16    929.6±8.21µs 67.2 MElem/sec
dict_intern_emit/intern_emit/size_8192_card_1000_null_0.00                        1.27    160.6±1.25µs 48.6 MElem/sec    1.00    126.3±0.71µs 61.9 MElem/sec
dict_intern_emit/intern_emit/size_8192_card_20_null_0.00                          1.00    104.5±1.15µs 74.8 MElem/sec    1.30  136.2±176.93µs 57.4 MElem/sec
dict_intern_emit/intern_emit/size_8192_card_300_null_0.00                         1.00    123.4±3.12µs 63.3 MElem/sec    1.08    133.5±2.37µs 58.5 MElem/sec
dict_intern_emit/intern_emit/size_8192_card_75_null_0.00                          1.00    110.3±1.33µs 70.9 MElem/sec    1.09    120.0±0.47µs 65.1 MElem/sec
dict_intern_emit/intern_emit/size_8192_card_8192_null_0.00                        2.71    681.6±4.80µs 11.5 MElem/sec    1.00    252.0±9.82µs 31.0 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_65536_card_1000_null_0.10     1.00      4.4±0.02ms 57.3 MElem/sec    1.04      4.5±0.03ms 55.1 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_65536_card_20_null_0.10       1.01      4.2±0.03ms 60.2 MElem/sec    1.00      4.1±0.05ms 60.8 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_65536_card_300_null_0.10      1.00      4.3±0.03ms 58.6 MElem/sec    1.00      4.3±0.03ms 58.5 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_65536_card_65536_null_0.10    1.54     17.9±1.61ms 14.0 MElem/sec    1.00     11.7±0.07ms 21.4 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_65536_card_75_null_0.10       1.01      4.2±0.03ms 58.9 MElem/sec    1.00      4.2±0.02ms 59.6 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_8192_card_1000_null_0.10      1.00    622.8±4.38µs 50.2 MElem/sec    1.05    654.3±1.58µs 47.8 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_8192_card_20_null_0.10        1.01    512.8±3.55µs 60.9 MElem/sec    1.00    509.7±3.27µs 61.3 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_8192_card_300_null_0.10       1.01    549.4±3.43µs 56.9 MElem/sec    1.00    545.2±2.38µs 57.3 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_8192_card_75_null_0.10        1.01    529.2±4.21µs 59.1 MElem/sec    1.00    522.7±2.93µs 59.8 MElem/sec
dict_repeated_intern_emit/repeated_intern_emit/size_8192_card_8192_null_0.10      1.41   1480.2±8.81µs 21.1 MElem/sec    1.00  1046.9±743.13µs 29.9 MElem/sec

Resource Usage

dictionary_group_values — base (merge-base)

Metric	Value
Wall time	295.1s
Peak memory	584.0 MiB
Avg memory	127.2 MiB
CPU user	228.7s
CPU sys	15.4s
Peak spill	0 B

dictionary_group_values — branch

Metric	Value
Wall time	340.1s
Peak memory	477.1 MiB
Avg memory	64.7 MiB
CPU user	226.2s
CPU sys	16.0s
Peak spill	0 B

File an issue against this benchmark runner

Rich-T-kid · 2026-07-07T18:40:54Z

+        let unique_lhs_row_count =
+            lhs_rows.iter().collect::<hashbrown::HashSet<_>>().len();
+        if unique_lhs_row_count <= rhs_rows.len() / CACHE_USE_THRESHOLD {
+            let mut comparison_cache = self.key_to_group_cache.lock().unwrap();
+            for (position, (&lhs_row, &rhs_row)) in
+                lhs_rows.iter().zip(rhs_rows).enumerate()
+            {
+                if !equal_to_results.get_bit(position) {
+                    continue;
+                }
+                let dict_key = dict.key(rhs_row);
+                let is_equal = *comparison_cache
+                    .entry((lhs_row, dict_key))
+                    .or_insert_with(|| match dict_key {
+                        None => self.inner.equal_to(lhs_row, &self.null_array, 0),
+                        Some(key) => self.inner.equal_to(lhs_row, dict_values, key),
+                    });


This is causing more overhead than its saving.

hashset insertions + mutext lock + hashsing two usize's
O(size * hash) O(1) O(~ unqiue values)

I think the best thing to do here to to avoid any regressions. That means making this implementation a thin wrapper over an inner GroupColumns implementation. a follow up PR can try to optimize the low-cardinality case similar to #21765

github-actions Bot added the physical-plan Changes to the physical-plan crate label Jun 25, 2026

Rich-T-kid added 5 commits June 29, 2026 18:43

introduce dictionarys as a supported group column type

e6b6dce

add schema support for dictionarys

8ce3f90

introduce high level GroupValuesColumn test

4dcc627

introduce groupColumn trait test

c3b71dd

git issues

1ffc7f0

Rich-T-kid force-pushed the rich-T-kid/dictionary-groupValuesColumn-impl branch from 2b60132 to 1ffc7f0 Compare June 30, 2026 20:23

Rich-T-kid added 4 commits June 30, 2026 16:30

introduce edge case/ regression section

3f7ff57

inital impl

d13e1a7

working implementation of dictionary for groupValuesCOlumns

14a4d40

benchmarks show perf boost over groupvaluerows, TODO:dedupe items bef…

52f620a

…ore inner append

Rich-T-kid force-pushed the rich-T-kid/dictionary-groupValuesColumn-impl branch from 3f7ff57 to e6b6dce Compare July 1, 2026 04:25

Rich-T-kid commented Jul 1, 2026

View reviewed changes

Comment thread datafusion/physical-plan/src/aggregates/group_values/multi_group_by/mod.rs

Rich-T-kid commented Jul 1, 2026

View reviewed changes

Comment thread datafusion/physical-plan/src/aggregates/group_values/multi_group_by/dictionary.rs Outdated

Rich-T-kid added 2 commits July 1, 2026 09:44

fix clippy errors & inline final builder

02ff545

add cache for arc ptr

26abc2c

trim down test

446e8d6

Rich-T-kid mentioned this pull request Jul 1, 2026

Test: add more aggregation focused dictionary sql logic test #23280

Open

Rich-T-kid force-pushed the rich-T-kid/dictionary-groupValuesColumn-impl branch from eb41915 to 770abfe Compare July 1, 2026 15:30

trim test LOC again

243a557

Rich-T-kid force-pushed the rich-T-kid/dictionary-groupValuesColumn-impl branch from 770abfe to 243a557 Compare July 1, 2026 15:32

Rich-T-kid changed the title ~~[Draft][22715] introduce dictionarys as a supported group column type~~ Feat:[Draft][22715] add dictionaries as a supported group column type Jul 1, 2026

Rich-T-kid force-pushed the rich-T-kid/dictionary-groupValuesColumn-impl branch from 152c1f0 to f3387c5 Compare July 5, 2026 03:59

speed up low cardinlaity case

09fcdef

Rich-T-kid mentioned this pull request Jul 6, 2026

feat: add GroupColumn support for List / Struct / Map / FixedSizeList in multi-column GROUP BY #22682

Open

working version

3d1e1c9

Rich-T-kid commented Jul 6, 2026

View reviewed changes

introduce inter-batch caching

ed0d612

Rich-T-kid commented Jul 6, 2026

View reviewed changes

Comment thread datafusion/physical-plan/src/aggregates/group_values/multi_group_by/dictionary.rs Outdated

break complex types into seperate parts

5f03e49

Rich-T-kid force-pushed the rich-T-kid/dictionary-groupValuesColumn-impl branch from f1391b9 to 5f03e49 Compare July 7, 2026 03:49

fixed breaking test, re-allocate hashtable on each intern() call

57d7b8c

re-introduce cache

894a0a6

Rich-T-kid force-pushed the rich-T-kid/dictionary-groupValuesColumn-impl branch from d4ea0f9 to 894a0a6 Compare July 7, 2026 17:32

Rich-T-kid commented Jul 7, 2026

View reviewed changes

Rich-T-kid marked this pull request as ready for review July 7, 2026 18:11

Rich-T-kid commented Jul 7, 2026

View reviewed changes

remove mutex

1f4816e

Rich-T-kid changed the title ~~Feat:[Draft][22715] add dictionaries as a supported group column type~~ Feat: add dictionaries as a supported group column type Jul 8, 2026

Uh oh!

Conversation

Rich-T-kid commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Rich-T-kid commented Jul 1, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Rich-T-kid commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

geoffreyclaude commented Jul 1, 2026

Uh oh!

adriangbot commented Jul 1, 2026

Uh oh!

adriangbot commented Jul 1, 2026

Uh oh!

Rich-T-kid commented Jul 1, 2026

Uh oh!

kumarUjjawal commented Jul 2, 2026

Uh oh!

Rich-T-kid commented Jul 6, 2026

Uh oh!

Rich-T-kid commented Jul 6, 2026

Uh oh!

geoffreyclaude commented Jul 6, 2026

Uh oh!

adriangbot commented Jul 6, 2026

Uh oh!

Rich-T-kid left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

adriangbot commented Jul 6, 2026

Uh oh!

Rich-T-kid commented Jul 6, 2026

Uh oh!

Uh oh!

Rich-T-kid commented Jul 7, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Rich-T-kid commented Jul 7, 2026

Uh oh!

geoffreyclaude commented Jul 7, 2026

Uh oh!

Rich-T-kid commented Jul 7, 2026

Uh oh!

geoffreyclaude commented Jul 7, 2026

Uh oh!

adriangbot commented Jul 7, 2026

Uh oh!

adriangbot commented Jul 7, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Rich-T-kid commented Jun 25, 2026 •

edited

Loading

Rich-T-kid commented Jul 1, 2026 •

edited

Loading