Skip to content

[Parquet] ALP encoder/decoder support - #9372

Merged
alamb merged 80 commits into
apache:mainfrom
sdf-jkl:alp
Sep 4, 2026
Merged

[Parquet] ALP encoder/decoder support#9372
alamb merged 80 commits into
apache:mainfrom
sdf-jkl:alp

Conversation

@sdf-jkl

@sdf-jkl sdf-jkl commented Feb 7, 2026 •
edited by alamb
Loading

Copy link
Copy Markdown
Member

Which issue does this PR close?

Rationale for this change

check issue

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

@github-actions github-actions Bot added the parquet Changes to the parquet crate label Feb 7, 2026
@alamb

alamb commented Feb 8, 2026

Copy link
Copy Markdown
Contributor

amaaazing

@sdf-jkl

sdf-jkl commented Feb 20, 2026

Copy link
Copy Markdown
Member Author

@alamb I worked with codex on replicating the c++ implementation reviewing commit by commit.

I'll do one final read myself, but this should be ready for initial review.

@alamb

alamb commented Feb 26, 2026

Copy link
Copy Markdown
Contributor

Thank you -- I will put this on my short list to reivew I was out last week

devanbenz
devanbenz suggested changes Feb 27, 2026
View reviewed changes

@devanbenz devanbenz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few comments

Comment thread parquet/src/encodings/decoding/alp.rs Outdated
Comment thread parquet/src/encodings/decoding/alp.rs Outdated
devanbenz
devanbenz reviewed Feb 27, 2026
View reviewed changes
Comment thread parquet/src/encodings/decoding/alp.rs Outdated
@sdf-jkl

sdf-jkl commented Mar 10, 2026

Copy link
Copy Markdown
Member Author

@devanbenz Sorry for taking so long to address your comments. I'll make sure to work on it this week!

@alamb

alamb commented Mar 11, 2026

Copy link
Copy Markdown
Contributor

The final spec is

devanbenz
devanbenz previously requested changes Mar 11, 2026
View reviewed changes

@devanbenz devanbenz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment about header regarding publication of spec.

Comment thread parquet/src/encodings/decoding/alp.rs Outdated
@sdf-jkl

sdf-jkl commented Mar 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Member Author

Thanks @devanbenz, I've finally addressed your reviews!

@devanbenz

devanbenz commented Mar 12, 2026

Copy link
Copy Markdown
Contributor

Thanks @devanbenz, I finally addressed your reviews!

Please cargo fmt the code 🫡

@sdf-jkl

sdf-jkl commented Mar 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Member Author

I swear I did 😿
Had to run

cargo fmt -p parquet --  --config skip_children=true `find ./parquet -name "*.rs" \! -name format.rs`

5e5c8d2

@alamb alamb mentioned this pull request Aug 14, 2026
Draft
@mmgeorge mmgeorge mentioned this pull request Aug 19, 2026
Draft
@sdf-jkl

sdf-jkl commented Aug 21, 2026

Copy link
Copy Markdown
Member Author

added #10785 to update the parquet-testing dep

@sdf-jkl

sdf-jkl commented Aug 21, 2026

Copy link
Copy Markdown
Member Author

Added another one with just the pinned rev change - #10786

sdf-jkl added 2 commits August 25, 2026 12:03
@sdf-jkl
Merge branch 'main' of https://github.com/apache/arrow-rs into alp
2d8002b 
# Conflicts:
#	parquet-testing
#	parquet/src/encodings/encoding/mod.rs
@sdf-jkl
clippy
baea4e9
@sdf-jkl

sdf-jkl commented Aug 25, 2026

Copy link
Copy Markdown
Member Author

@alamb :shipit:

@alamb

alamb commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

@Jefffrey and @etseidl -- do you think it is ok to merge the Rust ALP implementation for inclusion in arrow/parquet 60?

I think we could be one of the first major open source implementations to ship ALP if we did so

(thank you @sdf-jkl for all your work with this)

@alamb

alamb commented Aug 25, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

BTW if you want to see a sneak preview of a blog post about what ALP is

@sdf-jkl

sdf-jkl commented Aug 25, 2026

Copy link
Copy Markdown
Member Author

@alamb @devanbenz thanks for your help 🙏

@etseidl

etseidl commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

@Jefffrey and @etseidl -- do you think it is ok to merge the Rust ALP implementation for inclusion in arrow/parquet 60?

Yes, I don't see a problem with that. It's in "preview" in the spec, and it requires opt-in from users.

I think we could be one of the first major open source implementations to ship ALP if we did so

(thank you @sdf-jkl for all your work with this)

Yes thanks to all for getting this done so soon!

@sdf-jkl sdf-jkl mentioned this pull request Aug 25, 2026
Open
@Jefffrey

Jefffrey commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

sounds good; i was hoping to take a proper look at this since ive just skimmed it, but kept getting sidetracked by other PRs 😅

@alamb

alamb commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

sounds good; i was hoping to take a proper look at this since ive just skimmed it, but kept getting sidetracked by other PRs 😅

I think we still have some more time until the 60 release -- so we can wait a while to merge it in

@Jefffrey Jefffrey added enhancement Any new improvement worthy of a entry in the changelog api-change Changes to the arrow API labels Aug 31, 2026
@etseidl etseidl added the next-major-release the PR has API changes and it waiting on the next major version label Sep 1, 2026
@sdf-jkl sdf-jkl mentioned this pull request Sep 2, 2026
Merged
Jefffrey pushed a commit that referenced this pull request Sep 2, 2026
@sdf-jkl @cetra3
Add new parquet tests based on updated variant/json files (#10785)
2a1ba41 
# Which issue does this PR close?

- Follow-up to #10786, which updated the `parquet-testing` revision.
- No tracking issue yet; one can be added if needed.

# Rationale for this change

The updated `parquet-testing` revision included:

-
[apache/parquet-testing#113](apache/parquet-testing#113):
malformed and edge-case Variant files.
-
[apache/parquet-testing#117](apache/parquet-testing#117):
mark four shredded Variant cases as invalid because they omit required
`value` columns.
-
[apache/parquet-testing#118](apache/parquet-testing#118):
JSON and BSON logical-type files.
-
[apache/parquet-testing#119](apache/parquet-testing#119):
an extended ALP fixture.

#10786 made these fixtures available to Arrow Rust, but did not exercise
the JSON, BSON, or malformed Variant files.

The same revision also added an extended ALP fixture. That fixture is
covered by the ALP encoder/decoder work in #9372, where the required
decoding support exists.

# What changes are included in this PR?

- Read the JSON logical-type fixture and verify all decoded string
values.
- Read the BSON logical-type fixture and verify all decoded binary
values.
- Validate all 14 files under `bad_data/variants`, including the one
valid duplicate-offset case and 13 malformed cases.

The four shredded Variant cases renamed as invalid by the same
`parquet-testing` update are already exercised by the existing Variant
integration harness.

# Are these changes tested?

Yes. The Parquet and Variant integration tests pass locally, along with
formatting and clippy checks. The PR's CI checks are also green.

# Are there any user-facing changes?

No. This PR only adds integration-test coverage for existing
`parquet-testing` fixtures.

# AI usage

This PR was prepared with OpenAI Codex and reviewed by a human. The
integration tests, formatting, and clippy checks described above were
run against the final branch, and the GitHub CI checks passed.

---------

Co-authored-by: cetra3 <cetra3@hotmail.com>
@alamb

alamb commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

I plan to merge this PR tomorrow unless anyone would like more time to review beforehand

@alamb

alamb commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Ignored one new test for MIRI in ea6d049

Followed the same pattern as existing tests that try to use zstd

arrow-rs/parquet/src/compression.rs

Line 959 in ec21627

#[cfg_attr(miri, ignore)] // Zstd calls native C functions unsupported by Miri

@alamb

alamb commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

It is time.

Thank you (so much) @sdf-jkl and @devanbenz for all your work getting this into such great shape. I can't wait to see this released in 60.0.0 (hopefully next week)!

@alamb
alamb merged commit cb682a2 into apache:main Sep 4, 2026
31 checks passed
@sdf-jkl

sdf-jkl commented Sep 4, 2026

Copy link
Copy Markdown
Member Author

Thanks @alamb @devanbenz 🚀 🚀 🚀

@sdf-jkl sdf-jkl mentioned this pull request Sep 4, 2026
Open
Jefffrey pushed a commit that referenced this pull request Sep 5, 2026
@sdf-jkl
[Parquet] Alp criterion bench (#10662)
94464b4 
# Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax.
-->

- Closes #10642 .

- stacks on #9372 

# Rationale for this change

We need a benchmark to measure performance and prove future
optimizations
<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->

# What changes are included in this PR?

- Added a criterion benchmark
<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->

# Are these changes tested?

- N/A
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?

If this PR claims a performance improvement, please include evidence
such as benchmark results.
-->

# Are there any user-facing changes?

<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.

If there are any breaking changes to public APIs, please call them out.
-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alamb alamb alamb approved these changes
+1 more reviewer
@devanbenz devanbenz devanbenz approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

api-change Changes to the arrow API enhancement Any new improvement worthy of a entry in the changelog next-major-release the PR has API changes and it waiting on the next major version parquet Changes to the parquet crate

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Parquet] Prototype ALP encoding

5 participants

@sdf-jkl @alamb @devanbenz @etseidl @Jefffrey