Skip to content

feat(parquet): add ALP encoding for FLOAT and DOUBLE columns #1289

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
prtkgaur wants to merge 12 commits into
base: main
Choose a base branch
from
Open

feat(parquet): add ALP encoding for FLOAT and DOUBLE columns #1289

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
51c104d
Choose ALP scaling parameters by encoded size
sfc-gh-pgaur Oct 4, 2026
ce27eef
Encode and decode ALP float pages
ArnavBalyan Oct 4, 2026
65dc2da
Integrate ALP with Parquet readers and writers
sfc-gh-pgaur Oct 4, 2026
42d9c3b
Test ALP on a shared fixture and through Arrow
sfc-gh-pgaur Oct 4, 2026
a017cac
Benchmark ALP encoding and decoding
sfc-gh-pgaur Oct 4, 2026
d90a3da
Merge remote-tracking branch 'upstream/main' into alp-1289
zeroshade Oct 29, 2026
4fd6624
perf(parquet): hoist ALP scaling constants and add a vector size option
zeroshade Oct 29, 2026
9977f1b
fix(parquet): address ALP review findings
zeroshade Oct 29, 2026
1218254
fix(parquet/metadata): list RLE once when a data page used it
zeroshade Oct 29, 2026
a87963c
test(parquet): tighten and consolidate ALP tests
zeroshade Oct 29, 2026
2befc72
fix(parquet): estimate a buffered ALP vector at its all-exception size
zeroshade Oct 29, 2026
b7c6e12
fix(parquet): bound a buffered ALP vector's estimate by its largest size
zeroshade Oct 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion parquet-testing
Open in desktop
Submodule parquet-testing updated 45 files
+ − bad_data/ARROW-GH-47662.parquet
+30 −0 bad_data/README.md
+ − bad_data/variants/duplicate_field_offsets.parquet
+ − bad_data/variants/field_id_out_of_range.parquet
+ − bad_data/variants/int_overflow_in_bounds_check.parquet
+ − bad_data/variants/malformed_child_inside_well_formed_parent.parquet
+ − bad_data/variants/negative_dictionary_size.parquet
+ − bad_data/variants/out_of_range_child_offset.parquet
+ − bad_data/variants/out_of_range_dictionary_size.parquet
+ − bad_data/variants/out_of_range_element_count.parquet
+ − bad_data/variants/over_deep_nested_children.parquet
+ − bad_data/variants/oversized_primitive_size.parquet
+ − bad_data/variants/short_string_length_exceeds_buffer.parquet
+ − bad_data/variants/truncated_primitive_size.parquet
+ − bad_data/variants/unknown_primitive_type.parquet
+ − bad_data/variants/variant_version_2_header.parquet
+239 −0 data/README.md
+ − data/aes256/encrypt_columns_and_footer.parquet.encrypted
+ − data/aes256/encrypt_columns_and_footer_ctr.parquet.encrypted
+ − data/aes256/encrypt_columns_and_footer_disable_aad_storage.parquet.encrypted
+ − data/aes256/encrypt_columns_plaintext_footer.parquet.encrypted
+ − data/aes256/uniform_encryption.parquet.encrypted
+ − data/alp_extended.zstd.parquet
+ − data/bson.parquet
+ − data/encrypt_columns_and_footer_bloom_filter.parquet.encrypted
+14 −14 data/fixed_length_byte_array.md
+ − data/fixed_length_byte_array.parquet
+ − data/floating_orders_nan_count.parquet
+24 −0 data/geospatial/README.md
+ − data/geospatial/geography-lines.parquet
+ − data/geospatial/geography-points.parquet
+ − data/geospatial/geography-polygons.parquet
+185 −0 data/geospatial/geospatial-gen-geography.py
+77 −0 data/int96_timestamp_order.md
+ − data/int96_timestamp_order.parquet
+ − data/json.parquet
+ − shredded_variant/case-041-INVALID.parquet
+ − shredded_variant/case-041-INVALID_row-0.variant.bin
+ − shredded_variant/case-131-INVALID.parquet
+ − shredded_variant/case-131-INVALID_row-0.variant.bin
+ − shredded_variant/case-132-INVALID.parquet
+ − shredded_variant/case-132-INVALID_row-0.variant.bin
+ − shredded_variant/case-138-INVALID.parquet
+ − shredded_variant/case-138-INVALID_row-0.variant.bin
+12 −8 shredded_variant/cases.json
6 changes: 6 additions & 0 deletions parquet/doc.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,12 @@
//
// - Byte Stream Split (Float, Double, Int32, Int64, FixedLenByteArray)
//
// - ALP, Adaptive Lossless floating-Point (Float, Double). The Parquet
// format marks ALP as a preview encoding, so writing it is opt-in: enable
// it with WithALPEncoding, select it per column with WithEncodingFor or
// WithEncodingPath, and turn off dictionary encoding for those columns.
// Reading needs no option.
//
// Tip: Some platforms don't necessarily support all kinds of encodings. If you're not
// sure what to use, just use Plain and Dictionary encoding.
//
Expand Down
Loading
Loading