Fix test_remote_io.py due to mutating public s3 bucket#997
Closed
ShiboXing wants to merge 9 commits intometa-pytorch:mainfrom
Closed
Fix test_remote_io.py due to mutating public s3 bucket#997ShiboXing wants to merge 9 commits intometa-pytorch:mainfrom
ShiboXing wants to merge 9 commits intometa-pytorch:mainfrom
Conversation
Closed
ejguan
reviewed
Feb 8, 2023
Contributor
ejguan
left a comment
There was a problem hiding this comment.
Do you mind removing this test?
https://github.com/pytorch/data/blob/e14927f32a6cf99658f72b765425bf032116f724/test/test_remote_io.py#L337-L343
It takes too much time as it would download the whole archives. We have used the test to validated S3FileLoader for an image.
ejguan
reviewed
Feb 8, 2023
ejguan
reviewed
Feb 8, 2023
Contributor
ejguan
left a comment
There was a problem hiding this comment.
And, you don't need to rely on CI to execute those tests. You can run the test on your local env by
pytest -v test/test_remote_io.py -k test_fsspec_io_iterdatapipe
pytest -v test/test_remote_io.py -k test_s3_io_iterdatapipe
Author
|
Hi @ejguan, can you start the checks again? Added the recursive option. |
NivekT
reviewed
Feb 8, 2023
Author
Contributor
facebook-github-bot
pushed a commit
that referenced
this pull request
Feb 9, 2023
…reset request in the dispatching process (#994) Summary: ### Changes - ~Fix S3 Tests~ in #997 - Ignore AttributeError for ReadingService when `gc` gets involved - Add a reset_iterator_counter to dispatching process to guarantee that the datapipe is reset when all loops have received the request. Otherwise, datapipe can be reset in the middle of iteration of the other loops. - Remove reference of the `thread` from `Prefetcher`. This would prevent racing condition when both `finally` in generator and `reset` function are accessing the same `thread`. Pull Request resolved: #994 Reviewed By: NivekT Differential Revision: D43094081 Pulled By: ejguan fbshipit-source-id: 3065bc6a994cc0ff5047d9b71970abc4c8d3e369
Contributor
|
@ShiboXing Can you please do a |
added 9 commits
February 9, 2023 15:24
2ab0120 to
576457b
Compare
Author
|
@ejguan, rebased and force-pushed! |
Contributor
|
@NivekT has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
ejguan
approved these changes
Feb 9, 2023
Contributor
ejguan
left a comment
There was a problem hiding this comment.
Thanks for your effort to make TochData' CI green again
NivekT
approved these changes
Feb 9, 2023
Contributor
ShiboXing
pushed a commit
to ShiboXing/data
that referenced
this pull request
Feb 9, 2023
) Summary: Please read through our [contribution guide](https://github.com/pytorch/data/blob/main/CONTRIBUTING.md) prior to creating your pull request. - Note that there is a section on requirements related to adding a new DataPipe. Fixes meta-pytorch#984 - Add a private function to TestDataPipeRemoteIO in test_remote_io.py to get s3 objects count label through aws cli - add awscli in requirements Pull Request resolved: meta-pytorch#997 Reviewed By: ejguan Differential Revision: D43157757 Pulled By: NivekT fbshipit-source-id: 7e9ee8299a28a087f88024c3b3e77be3bfe5adf0
ShiboXing
pushed a commit
to ShiboXing/data
that referenced
this pull request
Feb 9, 2023
) Summary: Please read through our [contribution guide](https://github.com/pytorch/data/blob/main/CONTRIBUTING.md) prior to creating your pull request. - Note that there is a section on requirements related to adding a new DataPipe. Fixes meta-pytorch#984 - Add a private function to TestDataPipeRemoteIO in test_remote_io.py to get s3 objects count label through aws cli - add awscli in requirements Pull Request resolved: meta-pytorch#997 Reviewed By: ejguan Differential Revision: D43157757 Pulled By: NivekT fbshipit-source-id: 7e9ee8299a28a087f88024c3b3e77be3bfe5adf0
ShiboXing
pushed a commit
to ShiboXing/data
that referenced
this pull request
Feb 9, 2023
) Summary: Please read through our [contribution guide](https://github.com/pytorch/data/blob/main/CONTRIBUTING.md) prior to creating your pull request. - Note that there is a section on requirements related to adding a new DataPipe. Fixes meta-pytorch#984 - Add a private function to TestDataPipeRemoteIO in test_remote_io.py to get s3 objects count label through aws cli - add awscli in requirements Pull Request resolved: meta-pytorch#997 Reviewed By: ejguan Differential Revision: D43157757 Pulled By: NivekT fbshipit-source-id: 7e9ee8299a28a087f88024c3b3e77be3bfe5adf0
kace
added a commit
to kace/data
that referenced
this pull request
Feb 9, 2023
Fix test_remote_io.py due to mutating public s3 bucket (meta-pytorch#997)
ShiboXing
pushed a commit
to ShiboXing/data
that referenced
this pull request
Feb 10, 2023
) Summary: Please read through our [contribution guide](https://github.com/pytorch/data/blob/main/CONTRIBUTING.md) prior to creating your pull request. - Note that there is a section on requirements related to adding a new DataPipe. Fixes meta-pytorch#984 - Add a private function to TestDataPipeRemoteIO in test_remote_io.py to get s3 objects count label through aws cli - add awscli in requirements Pull Request resolved: meta-pytorch#997 Reviewed By: ejguan Differential Revision: D43157757 Pulled By: NivekT fbshipit-source-id: 7e9ee8299a28a087f88024c3b3e77be3bfe5adf0
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please read through our contribution guide prior to
creating your pull request.
Fixes #984
Changes