What is the problem the feature request solves?

After reviewing the memory management code in the native shuffle writer, there appear to be some bugs and/or inconsistencies, and there are no unit tests to prove that memory management is working as intended.

This issue is for re-implementing memory management along with unit tests.

Describe the potential solution

No response

Additional context

No response