Skip to content

Native shuffle double allocates memory #1448

@andygrove

Description

@andygrove

Describe the bug

As demonstrated in the new unit tests added in #1440, native shuffle is double allocating memory. This can contribute to excessive spilling.

        // TODO: note that we are currently double-counting the memory usage
        // because we reserve the memory twice - once at the repartitioner level
        // and then again in each PartitionBuffer
        assert_eq!(212992, repartitioner.reservation.size());
        assert_eq!(
            106496,
            repartitioner.buffered_partitions[0].reservation.size()
        );
        assert_eq!(
            106496,
            repartitioner.buffered_partitions[1].reservation.size()
        );

Steps to reproduce

No response

Expected behavior

No response

Additional context

No response

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions