Skip to content

fix: Reduce memory spike in aggregate window functions#15711

Open
zhli1142015 wants to merge 3 commits intofacebookincubator:mainfrom
zhli1142015:unboundframe
Open

fix: Reduce memory spike in aggregate window functions#15711
zhli1142015 wants to merge 3 commits intofacebookincubator:mainfrom
zhli1142015:unboundframe

Conversation

@zhli1142015
Copy link
Copy Markdown
Contributor

@zhli1142015 zhli1142015 commented Dec 5, 2025 •
edited
Loading

Reduce memory spike in aggregate window functions when using UNBOUNDED
PRECEDING AND UNBOUNDED FOLLOWING frame.

Previously, for window frames where all rows in a partition have the unbound frame
(e.g., UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING), the implementation would
allocate argument vectors for the entire partition at once via fillArgVectors.
For large partitions with millions of rows, this single massive allocation
causes a significant memory spike that can lead to OOM.

This change adds a dedicated sameFrameAggregation path that:

  1. Detects when all rows have identical frames by tracking allSameFrame in
    FrameMetadata.
  2. Processes data in batches (controlled by maxOutputBatchRows config)
    instead of allocating one giant vector for the entire partition. This
    significantly reduces peak memory usage.
  3. Caches the computed aggregate result after the first block, so subsequent
    output blocks can reuse the cached result without any data loading or
    recomputation.

The fix reduces memory spike for queries like:

SELECT sum(x) OVER (PARTITION BY p ORDER BY s
                    ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING)
FROM large_table  -- partition with millions of rows

Peak Memory Before (4.2 G)

      -- Window[2][STREAMING partition by [n0_1] order by [n0_2 ASC NULLS FIRST] t_10 := last_ignore_null("n0_0") ROWS between UNBOUNDED PRECEDING and UNBOUNDED FOLLOWING] -> n0_0:VARCHAR, n0_1:DATE, n0_2:INTEGER, t_10:VARCHAR
         Output: 100000000 rows (6.69GB, 24415 batches), Cpu time: 27.89s, Wall time: 28.08s, Blocked wall time: 0ns, Peak memory: 4.20GB, Memory allocations: 97685, Threads: 1, CPU breakdown: B/I/O/F (112.65ms/6.55s/21.10s/126.12ms)
            runningAddInputWallNanos     sum: 6.57s, count: 1, min: 6.57s, max: 6.57s
            runningFinishWallNanos       sum: 145.55ms, count: 1, min: 145.55ms, max: 145.55ms
            runningGetOutputWallNanos    sum: 21.16s, count: 1, min: 21.16s, max: 21.16s
            spillNotSupported            sum: 2, count: 2, min: 1, max: 1

Peak Memory After (2.7 G)

      -- Window[2][STREAMING partition by [n0_1] order by [n0_2 ASC NULLS FIRST] t_10 := last_ignore_null("n0_0") ROWS between UNBOUNDED PRECEDING and UNBOUNDED FOLLOWING] -> n0_0:VARCHAR, n0_1:DATE, n0_2:INTEGER, t_10:VARCHAR
         Output: 100000000 rows (6.69GB, 24415 batches), Cpu time: 26.81s, Wall time: 27.01s, Blocked wall time: 0ns, Peak memory: 2.70GB, Memory allocations: 97685, Threads: 1, CPU breakdown: B/I/O/F (116.28ms/7.48s/19.09s/127.72ms)
            runningAddInputWallNanos     sum: 7.51s, count: 1, min: 7.51s, max: 7.51s
            runningFinishWallNanos       sum: 146.44ms, count: 1, min: 146.44ms, max: 146.44ms
            runningGetOutputWallNanos    sum: 19.14s, count: 1, min: 19.14s, max: 19.14s
            spillNotSupported            sum: 2, count: 2, min: 1, max: 1

@netlify
Copy link
Copy Markdown

netlify Bot commented Dec 5, 2025 •
edited
Loading

✅ Deploy Preview for meta-velox canceled.

Name Link
🔨 Latest commit cfe1b37
🔍 Latest deploy log https://app.netlify.com/projects/meta-velox/deploys/69a232ebf6d63f00087fbe79

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Dec 5, 2025
@zhli1142015
Copy link
Copy Markdown
Contributor Author

zhli1142015 commented Dec 5, 2025

@mbasmanova and @aditi-pandit could you please help to review?

@zhli1142015
Copy link
Copy Markdown
Contributor Author

zhli1142015 commented Dec 8, 2025

CI failure is not related, #15697

@zhli1142015
Copy link
Copy Markdown
Contributor Author

zhli1142015 commented Dec 8, 2025

cc @JkSelf also.

@zhli1142015 zhli1142015 changed the title fix: Reduce memory spike in aggregate window functions with UNBOUNDED frame fix: Reduce memory spike in aggregate window functions Dec 12, 2025
@FelixYBW FelixYBW mentioned this pull request Feb 7, 2026
Open
@kgpai
Copy link
Copy Markdown
Contributor

kgpai commented Feb 10, 2026

cc: @kagamiori

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@zhli1142015 @kgpai