If the write destination is slow then the dataset writer should back off and apply backpressure to pause the reader. This will allow simple dataset API scans to operate on large out of core datasets.

This is dependent on ARROW-13611 which adds a backpressure feature for regular scanning (but not writing data) and on ARROW-13542 which moves the dataset write to a node in the exec plan.

Reporter: Weston Pace / @westonpace
Assignee: Weston Pace / @westonpace

Related issues:

• [C++] Scanning datasets does not enforce back pressure (depends upon)
• [C++][Compute][Dataset] Add dataset::WriteNode for writing rows from an ExecPlan to disk (depends upon)

PRs and other links:

• GitHub Pull Request #11286

_{Note: This issue was originally created as ARROW-14191. Please see the migration documentation for further details.}