variant: optimize variant object path access (79% lower latency) - #10358
variant: optimize variant object path access (79% lower latency)#10358friendlymatthew wants to merge 2 commits into
Conversation
|
cc @alamb |
alamb
left a comment
alamb
left a comment
There was a problem hiding this comment.
Looks pretty good overall -- thank you @friendlymatthew -- I had a few suggestions.
| .expect("created struct should be a valid shredded variant") | ||
| } | ||
|
|
||
| fn create_unshredded_object_variant_array(size: usize) -> VariantArray { |
There was a problem hiding this comment.
It would be helpful if you split the change to the benchmark off into a separate PR I can merge it and then use the benchmarking scripts
| builder.append_null()?; | ||
| false // add this to make match arms have the same return type | ||
|
|
||
| if has_path |
There was a problem hiding this comment.
Could you add some comments to explain the rationale for this condition / fast path?
| && target.typed_value_column().is_none() | ||
| && let Some(value_column) = target.value_column() | ||
| { | ||
| let metadata_column = target.metadata_column(); |
There was a problem hiding this comment.
Given that this function is already pretty long, could you please try and refactor some of the logic into smaller documented functions that each do some particular operation?
|
FYI @adriangb -- is this one related to your recent work? |
|
3 participants
Rationale for this change
Based on benchmark #10357
This PR speeds up field extraction from unshredded variant objects by avoiding repeated field name decoding for every row
Variant object keys are stored as numeric field ids that map to a metadata dictionary.
variant_getcurrently stores a named path by repeatedly mapping those ids back to strings and comparing the strings for each row, even when a batch shares one metadata dictionary. This repeated per-row work dominates when dealing with wide dictionariesThis optimization adds metadata scoped field id lookups, caches parsed metadata, and resolves named path elements to ids once per metadata dictionary before scanning rows. It re-resolves paths when the metadata changes and retains name lookup for unsorted dictionaries