Skip to content

Modify exception handling of online Datapipes #963

@SvenDS9

Description

@SvenDS9

🚀 The feature

Add an optional argument to HTTPReaderIterDataPipe to specify error handling.

https://github.com/pytorch/data/blob/2ca1fa6483e58c6428319393e1aab4c26f576bec/torchdata/datapipes/iter/load/online.py#L89-L94
_get_response_from_http() raises an exception if an error occurs.

Motivation, pitch

While working with HTTPReaderIterDataPipe I have noticed that failing HTTP-requests are not handled gracefully. This means that the DataPipe cannot be continued after an error occurs. Therefore I propose to add an optional argument which allows the user to continue the datapipe with the next url in case of an error.

Alternatives

See https://discuss.pytorch.org/t/exception-handling-when-using-datapipes/170959

Additional context

This also affects the other Datapipes in online.py which read from remote. It would probably make sense to add this feature there as well.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions