Dataset writing via exec plan (ARROW-13542) does not apply back pressure currently and will take up far more RAM than it should when writing a large dataset. The node should be applying back pressure. However, the preferred back pressure method (via scheduling) will need to wait for ARROW-13576.

Once those two are finished this can be studied in more detail. Also, the vm.dirty_ratio might be experimented with. In theory we should be applying our own back pressure and have no need of dirty pages. In practice, it may be more work than we want to tackle right now and we just let it do its thing.

Reporter: Weston Pace / @westonpace
Assignee: Weston Pace / @westonpace

Related issues:

• [C++] Scanning datasets does not enforce back pressure (duplicates)

_{Note: This issue was originally created as ARROW-13590. Please see the migration documentation for further details.}