[WIP] Add support for ALP encoding - #1289
Draft
prtkgaur wants to merge 10 commits into
Draft
Conversation
Implement the ALP (Adaptive Lossless floating-Point) pseudo-decimal encoding for FLOAT and DOUBLE columns: sampling to pick the exponent and factor per vector, exception handling for values that do not round-trip, and bit-packing of the resulting integers. Reference: https://dl.acm.org/doi/10.1145/3626717
Replace golang.org/x/xerrors with the standard errors package, matching the rest of parquet/internal/encoding and dropping a dependency the module no longer carries. Also list Encoding_ALP in knownEncodingValues so EncodingValues() yields it.
The parameter search returned the first exponent and factor that encoded
every value without an exception, which is rarely the cheapest pair: a
float64 column of two-decimal prices packed at 47 bits per value where 10
bits suffice, because scaling by 10^14 leaves the integers wide.
Score each combination the way the reference implementation does, as
n*bit_width(max-min) + numExceptions*(exactTypeBits+16) bits, skip a
combination that encodes fewer than two values, and break ties towards the
larger exponent and then the larger factor. Search a 256-value equidistant
sample of each vector instead of the whole vector, and stop the shortlisted
search after four scalings in a row fail to improve on the best.
Two other defects fell out of the rewrite:
- The shortlist was built from a map, so two runs over the same input
could choose different scalings and write different bytes. The
exponent and the factor now break count ties.
- Encoding scanned each vector twice, once to count exceptions and once
to encode, and allocated per vector on both the encode and the decode
path. One pass now encodes and collects exceptions together, and the
decoder reuses a single bit reader across a page's vectors.
The decoder also validates the exponent, the factor, the exception count,
the bit width, and every exception position, so a corrupt page reports an
error rather than reading outside the vector.
parquet-format marks ALP as a preview feature and recommends that writers keep preview encodings behind a flag, because a reader that does not implement one fails on the file rather than returning wrong values. WithAlpEncoding grants a writer permission to use ALP; it does not select the encoding, which a column still asks for through WithEncoding or WithEncodingFor. Asking for ALP without the flag panics, as asking for any unusable encoding does. The check runs once the properties are built rather than inside each option, so an option that selects ALP may be given before the one that allows it.
The read test decoded one file named by an environment variable, so it skipped in every run that did not set the variable, and it checked the metadata rather than the values. Read all eight ALP files that apache/parquet-testing#100 adds, each against the CSV of values it holds. Four are written by the C++ implementation and four by parquet-java, so a scaling or rounding difference in the Go decoder shows up as a wrong value instead of passing a round trip against itself.
A column writes a new data page every time its buffered size passes the
page size, and calls FlushValues for each one without resetting the
encoder in between. The ALP encoder kept the encoded vectors, the vector
sizes and the value count across that call, so the second page of a column
began with the first page's vectors and the third with both. Reading such
a file returned the wrong values, and nothing caught it because every test
wrote a single page. The C++ encoder resets its sink in the same place.
FlushValues now drops what it wrote. The exponent and factor shortlist
survives, because it describes the column rather than the page, and a
second page of the same values now encodes to the same bytes as the first.
Two smaller changes, no behaviour attached:
- alpIsBasicException and alpIsException were unreachable from the
encoder, which inlined the round trip in both hot paths and left the
helpers to the tests. One alpTryEncode, returning the encoded value
and whether the round trip held, now serves both paths and the tests.
- The decoder allocated its offset array, decoded vector and delta
buffer in SetData, so a column of many pages paid for them per page.
They grow instead.
A page that holds no values wrote no bytes at all, so an optional column whose page is entirely null failed to read. FlushValues now always writes the header, which is what the C++ implementation writes for such a page. A page claiming ALP for a column of some other physical type reached decoder traits that have no ALP case, and panicked. The reader rejects the page instead. Nothing this library writes produces one, so the page comes from another writer or from corruption. Decode could not terminate when the page header claimed more values than the ALP header describes. The last vector of a page is short whenever the count is not a whole number of vectors, so the copy moved nothing and the loop had nothing to advance. Decode now stops at the page's own value count and reports the disagreement, which the column reader needs: a read of nothing leaves it where it was, so it asks again. parquet/file/alp_test.go collects the file-level tests: the all-null page with PLAIN as its control, nulls scattered through a page, random doubles including NaNs and subnormals, data page v2, and the two malformed pages above.
The stub page reader kept returning the same page, so a reader that asked for a second one would see the first again. Both tests stop on an error before that happens, but a stub that ends the chunk cannot mislead a later change.
One file held the scaling arithmetic, the parameter search and the page format. Each now has its own file and owns the constants it uses: - alp_scaling.go turns a float into the integer ALP stores, and back. - alp_params.go picks the exponent and factor for a vector. - alp_encoding.go writes and reads the bytes of a page. Three simplifications came out of the split. The exponent and factor pair had three representations, so the shortlist and its counting map now use alpEncodingParams directly, which drops the uint32 packing and unpacking. alpBitWidth wrapped bits.Len64 without adding anything. The two Type methods held the same switch, which is now alpParquetType. A vector header size replaces the constant covering only its first three fields, so the decoder reads the whole header before validating it and advances once. Also document the encoder and decoder methods whose contract is not obvious from the name, and cover the widest packing a page can carry: 1024 values whose differences need all 64 bits.
A column asking for ALP while dictionary encoding stays on writes PLAIN pages: the dictionary takes precedence, and the writer's fallback installs a PLAIN encoder rather than the encoding the column asked for. Every ALP test pairs the request with WithDictionaryDefault(false), so name that requirement where a caller reads about the option.
prtkgaur
force-pushed
the
alp-go-encoding
branch
from
September 5, 2026 01:42
db9b5c8 to
6ce73f9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Co-Author: @arnav.balyan
Rationale for this change
What changes are included in this PR?
Are these changes tested?
Are there any user-facing changes?