Skip to content

Set max message size on parquet v3 reader#91737

Merged
al13n321 merged 3 commits intoClickHouse:masterfrom
arthurpassos:parquet_v3_reader_max_message_size
Dec 11, 2025
Merged

Set max message size on parquet v3 reader#91737
al13n321 merged 3 commits intoClickHouse:masterfrom
arthurpassos:parquet_v3_reader_max_message_size

Conversation

@arthurpassos
Copy link
Copy Markdown
Contributor

@arthurpassos arthurpassos commented Dec 8, 2025

Changelog category (leave one):

  • Improvement

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Set max message size on parquet v3 reader to avoid getting DB::Exception: apache::thrift::transport::TTransportException: MaxMessageSize reached

Documentation entry for user-facing changes

  • Documentation is written (mandatory for new features)

@arthurpassos
Copy link
Copy Markdown
Contributor Author

arthurpassos commented Dec 8, 2025

@al13n321 can you have a look?

@alexey-milovidov alexey-milovidov added the can be tested Allows running workflows for external contributors label Dec 8, 2025
@clickhouse-gh
Copy link
Copy Markdown
Contributor

clickhouse-gh bot commented Dec 8, 2025 •
edited
Loading

Workflow [PR], commit [ac7088d]

Summary: ❌

job_name test_name status info comment
Integration tests (amd_tsan, 1/6) failure
test_ttl_replicated/test.py::test_ttl_compatibility[node_left1-node_right1-1] FAIL cidb
BuzzHouse (amd_debug) failure
Logical error: 'Inconsistent AST formatting: the query: FAIL cidb
Integration tests (amd_tsan, 5/6) error

@clickhouse-gh clickhouse-gh bot added the pr-improvement Pull request with some product improvements label Dec 8, 2025
al13n321
al13n321 approved these changes Dec 10, 2025
View reviewed changes
Copy link
Copy Markdown
Member

@al13n321 al13n321 left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Over 100 MiB of FileMetaData? Wow. Is it mostly ColumnChunks for lots of columns in lots of row groups? How big is the file? Just curious.

@arthurpassos
Copy link
Copy Markdown
Contributor Author

arthurpassos commented Dec 10, 2025

Over 100 MiB of FileMetaData? Wow. Is it mostly ColumnChunks for lots of columns in lots of row groups? How big is the file? Just curious.

I need to double check (I don't have access to the file right now), but it was over 50GB

@arthurpassos @al13n321
Update src/Processors/Formats/Impl/Parquet/ThriftUtil.cpp
ac7088d 
Co-authored-by: Michael Kolupaev <michael.kolupaev@clickhouse.com>
@al13n321 al13n321 enabled auto-merge December 11, 2025 05:26
@al13n321 al13n321 added this pull request to the merge queue Dec 11, 2025
Merged via the queue into ClickHouse:master with commit adc8b08 Dec 11, 2025
126 of 130 checks passed
@robot-clickhouse-ci-2 robot-clickhouse-ci-2 added the pr-synced-to-cloud The PR is synced to the cloud repo label Dec 11, 2025
@arthurpassos arthurpassos mentioned this pull request Feb 6, 2026
Merged
25 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@al13n321 al13n321 al13n321 approved these changes

Assignees

@al13n321 al13n321

Labels

can be tested Allows running workflows for external contributors pr-improvement Pull request with some product improvements pr-synced-to-cloud The PR is synced to the cloud repo

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@arthurpassos @al13n321 @alexey-milovidov @robot-clickhouse-ci-2