-
Notifications
You must be signed in to change notification settings - Fork 174
Description
🚀 The feature
Add an optional argument to HTTPReaderIterDataPipe to specify error handling.
https://github.com/pytorch/data/blob/2ca1fa6483e58c6428319393e1aab4c26f576bec/torchdata/datapipes/iter/load/online.py#L89-L94
_get_response_from_http() raises an exception if an error occurs.
Motivation, pitch
While working with HTTPReaderIterDataPipe I have noticed that failing HTTP-requests are not handled gracefully. This means that the DataPipe cannot be continued after an error occurs. Therefore I propose to add an optional argument which allows the user to continue the datapipe with the next url in case of an error.
Alternatives
See https://discuss.pytorch.org/t/exception-handling-when-using-datapipes/170959
Additional context
This also affects the other Datapipes in online.py which read from remote. It would probably make sense to add this feature there as well.