GH-51238: [C++][Python][Parquet] Limit schema nesting depth when reading - #51239
Conversation
pitrou
added
backport-candidate
Critical Fix
a062326 to
76fba0f
Compare
| properties.set_footer_read_size( | ||
| parquet_scan_options->reader_properties->footer_read_size()); |
There was a problem hiding this comment.
This is a drive-by fix for an unrelated buglet.
github-actions
Bot
added
awaiting committer review
|
@github-actions crossbow submit -g cpp |
|
Revision: 131c58c Submitted crossbow builds: ursacomputing/crossbow @ actions-25d1b23663
|
|
@wgtmac @adamreeve @HuaHuaY Would you like to review this? |
| /// The default value is conservative enough for most use cases. | ||
| int32_t schema_depth_limit() const { return schema_depth_limit_; } | ||
| /// Set the schema nesting depth limit. | ||
| void set_schema_depth_limit(int32_t size) { schema_depth_limit_ = size; } |
There was a problem hiding this comment.
It was thinking if we need to reject a negative value here but it seems that it will safely throw later so I'm fine to leave it simple here.
There was a problem hiding this comment.
Do we have any conventions regarding the use of int32_t? Could we use uint32_t here?
There was a problem hiding this comment.
We use signed integers in most public APIs, we should probably not deviate here.
I can probably do it in this PR. |
131c58c to
8d4beae
Compare
8d4beae to
84d8a65
Compare
pitrou
changed the title
|
I have addressed all review comments (including the addition on the Python side), do you want to take another look? @adamreeve @wgtmac |
4 participants
Rationale for this change
Reconstructing a nested Schema from the Parquet Thrift metadata implies a recursive call that can blow up the stack on pathologically-nested schemas (with thousands of nesting levels or more).
By adding a limit on the schema nesting depth, we turn a stack overflow-induced crash into a regular Parquet error.
Are these changes tested?
By additional unit tests; also privately with a proof-of-concept reproducer that induces a stack overflow exhaustion.
Are there any user-facing changes?
In the unlikely case where a legitimate Parquet file has a deeper schema than the default schema nesting limit in this PR (100), an error will be raised when reading where it used to succeed. The user can bump the limit to circumvent the error.
This PR contains a "Critical Fix". It fixes a crash on a deeply nested Parquet schema that would provoke a stack overflow. It is not an exploitable vulnerability except through denial of service.
Thanks to "1K0CT" for the initial report.