GH-20127: [Python] Remove deprecated pyarrow.filesystem legacy implementations#39825
Merged
jorisvandenbossche merged 21 commits intoapache:mainfrom Mar 4, 2024
Merged
Conversation
Member
jorisvandenbossche
left a comment
There was a problem hiding this comment.
Looking good!
I assume there will be some declarations in libarrow.pxd that were used in _hdfsio.pyx and now are no longer used, and can be removed.
b479920 to
e005766
Compare
Member
|
Is this ready for review @AlenkaF ? |
Member
Author
|
I still need to fix one test which was failing and I added try-except (see #39825 (comment)) but I need to change the approach. Plan to work on it tomorrow! |
285d909 to
142af85
Compare
Member
Author
|
Ready for review @jorisvandenbossche @pitrou |
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Member
jorisvandenbossche
left a comment
There was a problem hiding this comment.
Looks good to me now! Just a few small nits
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
jorisvandenbossche
approved these changes
Mar 4, 2024
Member
|
The one failure on Windows is #40337 |
|
After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 2b194ad. There were no benchmark performance regressions. 🎉 The full Conbench report has more details. It also includes information about 1 possible false positive for unstable benchmarks that are known to sometimes produce them. |
jorisvandenbossche
added a commit
that referenced
this pull request
Mar 5, 2024
…sis setup (#40363) ### Rationale for this change Small follow-up on #39825, which removed the `test_hdfs.py` file itself, but didn't remove it from the hypothesis script * GitHub Issue: #20127 Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
npennequin
added a commit
to criteo/cluster-pack
that referenced
this pull request
Nov 14, 2024
The current implementation of EnhancedFileSystem is based on the legacy pyarrow filesystem interface that was removed in pyarrow 16.0.0 (apache/arrow#39825). We can entirely replace EnhancedFileSystem with fsspec. For HDFS fsspec relies on the new pyarrow filesystem interface. Behavior change note: for put, fsspec doesn't preserve file permissions Resolves #87
jcuquemelle
pushed a commit
to criteo/cluster-pack
that referenced
this pull request
Nov 21, 2024
The current implementation of EnhancedFileSystem is based on the legacy pyarrow filesystem interface that was removed in pyarrow 16.0.0 (apache/arrow#39825). We can entirely replace EnhancedFileSystem with fsspec. For HDFS fsspec relies on the new pyarrow filesystem interface. Behavior change note: for put, fsspec doesn't preserve file permissions Resolves #87
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR removes the
pyarrow.filesystemandpyarrow.hdfsfilesystems that have been deprecated since 2.0.0.