This is a collection of items to improve external (spilling) aggregation

Background

Abstract—Analytical database systems offer high-performance in-memory aggregation. If there are many unique groups, temporary query intermediates may not fit RAM, requiring the use of external storage. However, switching from an in-memory to an external algorithm can degrade performance sharply

• Robust External Hash Aggregation
  in the Solid State Age (DuckDB external aggregation paper))

DataFusion has supported memory limited / spilling hash aggregation since @kazuyukitanimura added it last year in #7400.

We can likely improve this feature and @2010YOUY01 is considering working on it

Tasks the solution you'd like

• Add benchmark for memory-limited aggregation #13090
• Optimized spill file format #14078
• Ensure GroupByHash does not error when trying to spill (calling try_resize where error is not acceptible) #14851