Skip to content

[Data] Deprecate read_parquet_bulk#48691

Merged
bveeramani merged 1 commit intomasterfrom
deprecate-read-parquet-bulk
Nov 12, 2024
Merged

[Data] Deprecate read_parquet_bulk#48691
bveeramani merged 1 commit intomasterfrom
deprecate-read-parquet-bulk

Conversation

@bveeramani
Copy link
Copy Markdown
Member

@bveeramani bveeramani commented Nov 11, 2024 •
edited
Loading

Why are these changes needed?

Users (including Ray Data developers!) are often confused about how to choose between read_parquet and read_parquet_bulk. To avoid confusion, this PR deprecates read_parquet_bulk.

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@bveeramani
Initial commit
8c51310 
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
omatthew98
omatthew98 reviewed Nov 12, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@omatthew98 omatthew98 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹

@bveeramani bveeramani enabled auto-merge (squash) November 12, 2024 18:33
@github-actions github-actions bot added the go add ONLY when ready to merge, run all tests label Nov 12, 2024
@bveeramani bveeramani merged commit 7bbab39 into master Nov 12, 2024
@bveeramani bveeramani deleted the deprecate-read-parquet-bulk branch November 12, 2024 19:23
JP-sDEV pushed a commit to JP-sDEV/ray that referenced this pull request Nov 14, 2024
@bveeramani @JP-sDEV
[Data] Deprecate read_parquet_bulk (ray-project#48691)
1d90b5b 
Users (including Ray Data developers!) are often confused about how to
choose between `read_parquet` and `read_parquet_bulk`. To avoid
confusion, this PR deprecates `read_parquet_bulk`.

Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@JackGammack
Copy link
Copy Markdown
Contributor

JackGammack commented Jul 2, 2025

Is there more information about why read_parquet_bulk was deprecated? Or recommendations for how to read many thousands of parquet files that already have the same schema without the huge overhead/startup time of read_parquet? This startup time can take 30+ minutes with a very large number of files.

Using FastFileMetadataProvider works for some other datasources, but fails with read_parquet:
AttributeError: 'FastFileMetadataProvider' object has no attribute 'prefetch_file_metadata'

@alexeykudinkin
Copy link
Copy Markdown
Contributor

alexeykudinkin commented Jul 2, 2025

Hey, @JackGammack! We're currently looking into addressing some of the long-standing issues with reading Parquet

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@omatthew98 omatthew98 omatthew98 approved these changes

@scottjlee scottjlee Awaiting requested review from scottjlee

@raulchen raulchen Awaiting requested review from raulchen

@stephanie-wang stephanie-wang Awaiting requested review from stephanie-wang

@alexeykudinkin alexeykudinkin Awaiting requested review from alexeykudinkin

@srinathk10 srinathk10 Awaiting requested review from srinathk10

Assignees

@omatthew98 omatthew98

Labels

community-backlog go add ONLY when ready to merge, run all tests

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@bveeramani @JackGammack @alexeykudinkin @omatthew98 @hainesmichaelc