🚀 The feature

Currently, if any Error happens in the sub-process, the child process would just exits without passing Error message back to the main process. And, the data pipeline in the main process will continue without proper exit.
In order to achieve this feature, we need to do the following step:

• Add a way to pass Error message
  • From worker process to main process via res_queue Forward worker exceptions and exit with it #1003
  • From dispatching process to worker process then to main process via res_queue
  • Add traceback for worker exception
    • Give name of _IterateQueueDataPipes to differentiate Error from worker or dispatching process
• Handle Error
  • Properly exit other worker process when one worker sends Error message to main process
  • Properly exit worker processes when the Error comes from the dispatching process, then exit main process

Motivation, pitch

Increase reliability of DataLoader with PrototypeMultiprocessingReadingService

Alternatives

No response

Additional context

No response