Invitation to provide feedback on New Parquet Encoding: Adaptive Lossless Floating Point Compression

[alamb]

Describe the enhancement requested

👋 from the Parquet community

The Parquet community is working on a proposal to implement ALP: Adaptive Lossless floating-Point Compression, (SIGMOD 2024) as a new Parquet encoding

arrow-go is listed on the Parquet Implementation Status page

Since you may feel pressure to implement newly added encodings, I think having your feedback on the spec as a stakeholder would be immensely valuable.

The proposed addition is here : Parquet ALP Encoding Specification (Google Doc)

There is also a prototype C++ implementation for review here

• GH-48701: [C++][Parquet] Add ALPpd encoding arrow#48345

See also: mailing list discussion (link) and parquet-format PR

Thank you for your consideration

Component(s)

Parquet

[zeroshade]

Thanks @alamb for the nudge here!

I'll take a look at the format PR and the prototype implementation and provide feedback. I should be able to find some time this week to do so. It'll give me something to do under all this snow ❄️ 😆

[alamb]

Note that the parquet-format PR is out of date with respect to the google doc spec -- please give me a few more days to migrate the content over to parquet format (or use the google doc spec directly)

[alamb]

(the basic algorithm is the same, but the details of where the headers go is still being worked out)

[alamb]

FWI we have merged the spec

• [Proposal] Add ALP encoding support in parquet file format parquet-format#533

As well the parquet-testing repo now has a example file

• Add single ALP test file (216KB, covers corner cases) parquet-testing#119

[zeroshade]

The implementation of ALP here is currently in #704 and waiting on @prtkgaur and @ArnavBalyan (or anyone who wants to pick it up) to continue responding to the feedback there and bring it across the finish line.