Skip to content

Should we deprecate chunksize from read_parquet? #9043

@rjzamora

Description

@rjzamora

Follow-up to recent issues like #8937

Recent PRs have simplified the read_parquet API a bit, but the code is still vast. After some careful consideration, I'd like to suggest that we deprecate the chunksize option.

Why to deprecate chunksize:

  • No one seems to be using it (Please do correct me if I'm wrong!)
  • The current implementation is not very efficient (and is extremely slow on remote file systems)
  • There is no case where this feature is required for stability (probably why it is never used)

cc @jcrist

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions