-
Notifications
You must be signed in to change notification settings - Fork 270
Closed
Description
Describe the bug
As demonstrated in the new unit tests added in #1440, native shuffle is double allocating memory. This can contribute to excessive spilling.
// TODO: note that we are currently double-counting the memory usage
// because we reserve the memory twice - once at the repartitioner level
// and then again in each PartitionBuffer
assert_eq!(212992, repartitioner.reservation.size());
assert_eq!(
106496,
repartitioner.buffered_partitions[0].reservation.size()
);
assert_eq!(
106496,
repartitioner.buffered_partitions[1].reservation.size()
);Steps to reproduce
No response
Expected behavior
No response
Additional context
No response