Repository navigation
Conversation
prtkgaur
force-pushed
the
alp-go-encoding
branch
from
September 5, 2026 01:42
db9b5c8 to
6ce73f9
Compare
prtkgaur
force-pushed
the
alp-go-encoding
branch
2 times, most recently
from
October 4, 2026 02:39
22e6bde to
44ffbd8
Compare
ALP turns a float into an integer by multiplying it by 10^exponent and then by 10^-factor. This adds that arithmetic and the search that picks the pair for each vector. The search estimates the size of every pair on a sample of 256 values, counting the packed width of the values that convert and the full cost of those that do not, and breaks ties toward the larger exponent as the ALP paper does. After eight vectors it only tries the five pairs those vectors chose most often. The arithmetic follows the reference implementation's rounding and order of multiplication exactly. Rounding decides which values become exceptions, so a writer that differs by one bit writes a different page.
This adds the ALP page format for FLOAT and DOUBLE columns. A page has a seven-byte header, one offset per vector, and vectors of 1024 values. Each vector stores its scaling, a frame of reference, the bit-packed differences from that frame, and the values that did not survive the scaling as exceptions patched in by position. Packing uses the existing BitWriter and BitReader. The differences are computed in unsigned arithmetic, so a DOUBLE vector spanning most of the int64 range packs at 64 bits instead of overflowing. The decoder accepts any vector size a page declares, from 8 to 32768 values, so it reads pages from writers that do not use 1024. It checks every offset, bit width and exception position against the page before reading, and returns an error when the page holds fewer values than the page header claims instead of a short read the column reader would retry forever. ALP takes encoding id 10. The generated Thrift bindings have no checked-in source, so the new value is added to them by hand. Co-authored-by: Prateek Gaur <prateek.gaur@snowflake.com>
parquet-format marks ALP as a preview encoding, and a reader without it cannot open a file that uses it. Writing ALP therefore takes an explicit opt-in, WithALPEncoding for every column or WithALPEncodingFor for one, and NewWriterProperties panics when a column asks for ALP without it, as it does for any unusable encoding. The opt-in permits ALP but does not select it. A column still chooses it with WithEncoding or WithEncodingFor, and needs dictionary encoding off, because a writer that abandons its dictionary falls back to PLAIN. The column reader returns an error for an ALP page on a column that is not FLOAT or DOUBLE, where the decoder traits would otherwise panic. The tests write and read columns of several pages with and without nulls, under both data page versions.
A round trip through one implementation cannot show that it reads what other writers produce. This moves parquet-testing to the commit that adds alp_extended.zstd.parquet, whose ALP columns use vectors of 32, 1024 and 4096 values next to PLAIN copies of the same data, and checks that every ALP value decodes to the bits of its PLAIN copy. A second test writes an Arrow table with ALP columns and reads it back across compression codecs, several pages and row groups, nulls, and values that can only be stored as exceptions.
The exception rate drives what ALP costs, so the benchmarks encode and decode FLOAT and DOUBLE columns at several sizes with no exceptions, one exception in ten values, and nothing but exceptions. A batched DOUBLE decode reads 137 values at a time, so batches end partway through a vector the way column reader calls do.
prtkgaur
force-pushed
the
alp-go-encoding
branch
from
October 8, 2026 03:33
44ffbd8 to
a017cac
Compare
# Conflicts: # parquet/file/file_reader_test.go
- Look the power-of-ten constants up once per vector (alpScaling) instead of once per value; encode is ~35-40% and decode ~30-44% faster. - Add the frame of reference back in the encoded integer's width, as the spec requires for FLOAT. - Reuse scratch for PutSpaced and the page header. - Add WithALPVectorSize/WithALPVectorSizeFor/WithALPVectorSizePath (power of two, 8-32768, default 1024) and apply it in the float column writers. - Document ALP in the package docs.
zeroshade
changed the title
- Treat a FLOAT value whose rounded integer leaves int32 as an exception. Rounding a scaled value just under 2^31 could reach 2^31, which readers rebuild in int32 with the sign flipped (or a 33-bit packing width). - Avoid int overflow on 32-bit platforms when counting vectors and bounds-checking vector offsets in the decoder. - List the encodings the data pages actually used in the column chunk metadata when there is no dictionary, so a column that asked for ALP but fell back to PLAIN is not labelled ALP. - Point the docs at per-column ALP selection, make ALPEncodingEnabled report the default like the other getters, and add ALPVectorSize. - Count vector offsets in EstimatedDataEncodedSize; fix stale comments.
- Cover V2 pages with nulls and compression, and the spaced write/read paths, which the existing tests named but did not reach. - Prove pages are ALP via data-page encoding stats, and check the many-pages test really writes several pages. - Merge the float32/float64 copy-paste round trips into one generic table test; drop tests that only restated types or duplicated the shared encoding suite; move reader-error tests to column_reader_test.go on testutils.MockPageReader; gather file-level ALP tests in alp_test.go. - Use checked allocators, and report allocations in the benchmarks.
EstimatedDataEncodedSize counted buffered values at their raw width, so a vector of NaNs or infinities (each stored with a 2-byte position, plus the vector header and offset) was underestimated and pages could overshoot the page size.
A vector can pack every slot at full width and also store exceptions, so sizing the pending vector as all exceptions still undercounted mixed vectors of NaNs and wide integers. Count both.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Co-Author: @arnav.balyan
Supersedes #704 (this PR started from it and addresses the review there).
Rationale for this change
ALP (Adaptive Lossless floating-Point) is the
ALP = 10encoding for FLOAT and DOUBLE columns in parquet-format (in Preview since 2026-08-01). It scales values by powers of ten into integers, then stores them frame-of-reference + bit-packed, keeping values that don't round-trip as exceptions. Arrow C++ already implements it.What changes are included in this PR?
Encoding (
parquet/internal/encoding)alp_encoding.go: genericalpEncoder[T]/alpDecoder[T]for float32 and float64, following the spec's page layout (7-byte header, offset array, self-describing vectors).FlushValues.log_vector_size(8–32768), and bounds-checks every size and offset read from the page.alp_scaling.go: the normative decodeencoded * 10^f * 10^-ewith the spec's literal power-of-ten tables.alpScaling[T]looks the constants up once per vector instead of once per value.alp_params.go: picks(exponent, factor)by estimated encoded size, using the paper's sampling scheme: 256 samples per vector, a full search over the first 8 vectors, then the 5 best combinations.utils.BitWriter/BitReaderfor bit packing.Writer/reader integration
parquet.Encodings.ALP, andEncoding_ALPin the generated thrift code.WithALPEncoding/WithALPEncodingFor/WithALPEncodingPathallow ALP, andWithEncoding*(Encodings.ALP)selects it.NewWriterProperties.WithALPVectorSize/WithALPVectorSizeFor/WithALPVectorSizePathset the vector size: a power of two from 8 to 32768, default 1024, matching thealp_vector_sizeproperty of the Arrow C++ implementation.Are these changes tested?
Yes.
TestReadALPConformanceFilereadsalp_extended.zstd.parquetfrom apache/parquet-testing, written by Arrow C++ with 32-, 1024- and 4096-value vectors. It bit-compares every ALP column against its PLAIN copy. Theparquet-testingsubmodule is bumped to include that file.BenchmarkALP{En,De}codingFloat{32,64}, plus batched decoding.Are there any user-facing changes?
Yes, all opt-in:
parquet.Encodings.ALPWithALPEncoding*andWithALPVectorSize*writer properties, with the matching gettersDefaultALPEnabledandDefaultALPVectorSizeReading ALP needs no option.