Improve the performance of pipeline building by Algunenano · Pull Request #83631 · ClickHouse/ClickHouse

Algunenano · 2025-07-11T15:05:55Z

Changelog category (leave one):

Performance Improvement

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Improve the performance of pipeline building

This PR generalizes the change from #70105 across the whole codebase, reducing the block copies by using shared pointers, aliased to SharedHeader as it's mostly used to keep reference to headers in the pipeline.

This change is tricky and I expect bugs to come out of it because Block::operator bool() exists and it's used frequently, and std::shared_ptr<Block>::operator bool() also exists and it's completely different meaning. Similar issue with the default constructor of both. I'm 100% sure I've missed some places, because I've already found and fixed a few.

Benchmarks

See perf tests. Depends on the query and join algorithm (different pipeline)

Pending

Pass tests. I expect several crashes and broken tests iteration before this is mergeable.
Add a new performance check

Documentation entry for user-facing changes

Documentation is written (mandatory for new features)

…_memcpy_cont

clickhouse-gh · 2025-07-11T15:06:21Z

Workflow [PR], commit [d1cb791]

Summary: ❌

job_name	test_name	status
Stateless tests (amd_asan, distributed plan, 2/2)		failure
	03221_insert_timeout_overflow_mode	FAIL
Stateless tests (amd_coverage,1/6)		dropped
Stateless tests (amd_coverage,2/6)		dropped
Stateless tests (amd_coverage,3/6)		dropped
Stateless tests (amd_coverage,4/6)		dropped
Stateless tests (amd_coverage,5/6)		dropped
Stateless tests (amd_coverage,6/6)		dropped
Performance Comparison (amd_release, master_head, 1/3)		error

…_memcpy_cont

src/Interpreters/InsertDependenciesBuilder.cpp

src/Processors/QueryPlan/AggregatingStep.cpp

…_memcpy_cont

Fix rebase issue: - 20250717 ClickHouse#83631 - 20250529 ClickHouse#80777 - 20250326 ClickHouse#77729 - 20250308 ClickHouse#77269

* [GLUTEN-1632][CH]Daily Update Clickhouse Version (20250720) * Fix gtest due to ClickHouse/ClickHouse#83599 * fxi build due to ClickHouse/ClickHouse#83631 * Fix Build due to ClickHouse/ClickHouse#83294 * Fix Build due to ClickHouse/ClickHouse#82881 --------- Co-authored-by: kyligence-git <gluten@kyligence.io> Co-authored-by: Chang chen <chenchang@apache.com>

Fix rebase issue: - 20250717 ClickHouse#83631 - 20250529 ClickHouse#80777 - 20250326 ClickHouse#77729 - 20250308 ClickHouse#77269

Fix rebase issue: - 20250717 ClickHouse#83631 - 20250529 ClickHouse#80777 - 20250326 ClickHouse#77729 - 20250308 ClickHouse#77269 (cherry picked from commit 4aeddc5)

Fix rebase issue: - 20250717 ClickHouse#83631 - 20250529 ClickHouse#80777 - 20250326 ClickHouse#77729 - 20250308 ClickHouse#77269 (cherry picked from commit 4aeddc5) (cherry picked from commit d2c3ee8)

Algunenano added 8 commits July 10, 2025 00:08

Extend the use of SharedHeader across the codebase

d87ee15

Fix warning

bd45820

Fix one crash

f24876c

Fix another crash

514aaab

Fix a bug

20a7039

Fix more bugs

c9597dc

Merge remote-tracking branch 'blessed/master' into reduce_port_header…

443bc6f

…_memcpy_cont

A bunch of minor optimizations for pipeline generation

78528b6

Algunenano self-assigned this Jul 11, 2025

clickhouse-gh bot added the pr-performance Pull request with some performance improvements label Jul 11, 2025

Algunenano added 6 commits July 11, 2025 17:43

Fix debug build and cleanup some includes

3b6270e

Style

59a0399

Tidy happy, engineer happy

fdf9980

Fix tidy

0273a4b

Merge remote-tracking branch 'blessed/master' into reduce_port_header…

ba8bae9

…_memcpy_cont

Add perf test

cad6e2a

Algunenano force-pushed the reduce_port_header_memcpy_cont branch from e4c9713 to cad6e2a Compare July 14, 2025 08:45

Less header copies

b3031e0

Algunenano mentioned this pull request Jul 14, 2025

ConcurrentHashJoin: Slow initialization and destruction #83703

Closed

Algunenano added 4 commits July 14, 2025 12:44

Prevent unnecessary checks

d2f6aa6

fix

e0f6056

Remove Block operator bool

6accf55

Merge remote-tracking branch 'blessed/master' into reduce_port_header…

5987ead

…_memcpy_cont

Algunenano commented Jul 14, 2025

View reviewed changes

src/Interpreters/InsertDependenciesBuilder.cpp Outdated Show resolved Hide resolved

Algunenano added 2 commits July 14, 2025 15:50

Self review

9c5a979

Fix debug build and minor review changes

e78ea54

Algunenano commented Jul 14, 2025

View reviewed changes

src/Processors/QueryPlan/AggregatingStep.cpp Show resolved Hide resolved

Algunenano added 2 commits July 15, 2025 09:58

Minor

4debd54

Merge remote-tracking branch 'blessed/master' into reduce_port_header…

7808fb1

…_memcpy_cont

baibaichen pushed a commit to Kyligence/gluten that referenced this pull request Jul 18, 2025

fxi build due to ClickHouse/ClickHouse#83631

d96cd86

baibaichen added a commit to Kyligence/ClickHouse that referenced this pull request Jul 18, 2025

Optimize sort spill (#490)

2fdb348

Fix rebase issue: - 20250717 ClickHouse#83631 - 20250529 ClickHouse#80777 - 20250326 ClickHouse#77729 - 20250308 ClickHouse#77269

baibaichen pushed a commit to Kyligence/gluten that referenced this pull request Jul 18, 2025

fxi build due to ClickHouse/ClickHouse#83631

a5a078c

baibaichen added a commit to Kyligence/ClickHouse that referenced this pull request Jul 20, 2025

Optimize sort spill (#490)

fd508b3

Fix rebase issue: - 20250717 ClickHouse#83631 - 20250529 ClickHouse#80777 - 20250326 ClickHouse#77729 - 20250308 ClickHouse#77269

baibaichen pushed a commit to Kyligence/gluten that referenced this pull request Jul 20, 2025

fxi build due to ClickHouse/ClickHouse#83631

7423538

kyligence-git pushed a commit to Kyligence/ClickHouse that referenced this pull request Jul 20, 2025

Optimize sort spill (#490)

75261a6

Fix rebase issue: - 20250717 ClickHouse#83631 - 20250529 ClickHouse#80777 - 20250326 ClickHouse#77729 - 20250308 ClickHouse#77269

kyligence-git pushed a commit to Kyligence/ClickHouse that referenced this pull request Jul 21, 2025

Optimize sort spill (#490)

1ee3907

Fix rebase issue: - 20250717 ClickHouse#83631 - 20250529 ClickHouse#80777 - 20250326 ClickHouse#77729 - 20250308 ClickHouse#77269

baibaichen added a commit to Kyligence/ClickHouse that referenced this pull request Jul 28, 2025

Optimize sort spill (#490)

c2cbb28

Fix rebase issue: - 20250717 ClickHouse#83631 - 20250529 ClickHouse#80777 - 20250326 ClickHouse#77729 - 20250308 ClickHouse#77269

baibaichen added a commit to Kyligence/ClickHouse that referenced this pull request Jul 29, 2025

Optimize sort spill (#490)

40078c1

Fix rebase issue: - 20250717 ClickHouse#83631 - 20250529 ClickHouse#80777 - 20250326 ClickHouse#77729 - 20250308 ClickHouse#77269

kyligence-git pushed a commit to Kyligence/ClickHouse that referenced this pull request Jul 29, 2025

Optimize sort spill (#490)

a520c99

Fix rebase issue: - 20250717 ClickHouse#83631 - 20250529 ClickHouse#80777 - 20250326 ClickHouse#77729 - 20250308 ClickHouse#77269

kyligence-git pushed a commit to Kyligence/ClickHouse that referenced this pull request Jul 30, 2025

Optimize sort spill (#490)

3553aa8

Fix rebase issue: - 20250717 ClickHouse#83631 - 20250529 ClickHouse#80777 - 20250326 ClickHouse#77729 - 20250308 ClickHouse#77269

kyligence-git pushed a commit to Kyligence/ClickHouse that referenced this pull request Jul 31, 2025

Optimize sort spill (#490)

3907340

Fix rebase issue: - 20250717 ClickHouse#83631 - 20250529 ClickHouse#80777 - 20250326 ClickHouse#77729 - 20250308 ClickHouse#77269

kyligence-git pushed a commit to Kyligence/ClickHouse that referenced this pull request Aug 1, 2025

Optimize sort spill (#490)

8b1a98b

Fix rebase issue: - 20250717 ClickHouse#83631 - 20250529 ClickHouse#80777 - 20250326 ClickHouse#77729 - 20250308 ClickHouse#77269

kyligence-git pushed a commit to Kyligence/ClickHouse that referenced this pull request Aug 2, 2025

Optimize sort spill (#490)

85f560a

Fix rebase issue: - 20250717 ClickHouse#83631 - 20250529 ClickHouse#80777 - 20250326 ClickHouse#77729 - 20250308 ClickHouse#77269

kyligence-git pushed a commit to Kyligence/ClickHouse that referenced this pull request Aug 3, 2025

Optimize sort spill (#490)

085e794

Fix rebase issue: - 20250717 ClickHouse#83631 - 20250529 ClickHouse#80777 - 20250326 ClickHouse#77729 - 20250308 ClickHouse#77269

baibaichen added a commit to Kyligence/ClickHouse that referenced this pull request Aug 7, 2025

Optimize sort spill (#490)

e2f5ae2

Fix rebase issue: - 20250717 ClickHouse#83631 - 20250529 ClickHouse#80777 - 20250326 ClickHouse#77729 - 20250308 ClickHouse#77269

baibaichen added a commit to Kyligence/ClickHouse that referenced this pull request Aug 8, 2025

Optimize sort spill (#490)

8814755

Fix rebase issue: - 20250717 ClickHouse#83631 - 20250529 ClickHouse#80777 - 20250326 ClickHouse#77729 - 20250308 ClickHouse#77269

baibaichen added a commit to Kyligence/ClickHouse that referenced this pull request Aug 11, 2025

Optimize sort spill (#490)

eeb4b41

Fix rebase issue: - 20250717 ClickHouse#83631 - 20250529 ClickHouse#80777 - 20250326 ClickHouse#77729 - 20250308 ClickHouse#77269

baibaichen added a commit to Kyligence/ClickHouse that referenced this pull request Aug 12, 2025

Optimize sort spill (#490)

18b4e4c

Fix rebase issue: - 20250717 ClickHouse#83631 - 20250529 ClickHouse#80777 - 20250326 ClickHouse#77729 - 20250308 ClickHouse#77269

baibaichen added a commit to Kyligence/ClickHouse that referenced this pull request Aug 13, 2025

Optimize sort spill (#490)

1ae6c02

Fix rebase issue: - 20250717 ClickHouse#83631 - 20250529 ClickHouse#80777 - 20250326 ClickHouse#77729 - 20250308 ClickHouse#77269

baibaichen added a commit to Kyligence/ClickHouse that referenced this pull request Aug 14, 2025

Optimize sort spill (#490)

78b2307

Fix rebase issue: - 20250717 ClickHouse#83631 - 20250529 ClickHouse#80777 - 20250326 ClickHouse#77729 - 20250308 ClickHouse#77269

baibaichen added a commit to Kyligence/ClickHouse that referenced this pull request Aug 15, 2025

Optimize sort spill (#490)

3e8d877

Fix rebase issue: - 20250717 ClickHouse#83631 - 20250529 ClickHouse#80777 - 20250326 ClickHouse#77729 - 20250308 ClickHouse#77269

baibaichen added a commit to Kyligence/ClickHouse that referenced this pull request Aug 16, 2025

Optimize sort spill (#490)

f078b10

Fix rebase issue: - 20250717 ClickHouse#83631 - 20250529 ClickHouse#80777 - 20250326 ClickHouse#77729 - 20250308 ClickHouse#77269

baibaichen added a commit to Kyligence/ClickHouse that referenced this pull request Aug 18, 2025

Optimize sort spill (#490)

394c62c

Fix rebase issue: - 20250717 ClickHouse#83631 - 20250529 ClickHouse#80777 - 20250326 ClickHouse#77729 - 20250308 ClickHouse#77269

lgbo-ustc pushed a commit to bigo-sg/ClickHouse that referenced this pull request Sep 17, 2025

Optimize sort spill (ClickHouse#490)

4aeddc5

Fix rebase issue: - 20250717 ClickHouse#83631 - 20250529 ClickHouse#80777 - 20250326 ClickHouse#77729 - 20250308 ClickHouse#77269

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve the performance of pipeline building#83631

Improve the performance of pipeline building#83631
Algunenano merged 25 commits intoClickHouse:masterfrom
Algunenano:reduce_port_header_memcpy_cont

Algunenano commented Jul 11, 2025 •

edited

Loading

Uh oh!

clickhouse-gh bot commented Jul 11, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Algunenano commented Jul 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Benchmarks

Pending

Documentation entry for user-facing changes

Uh oh!

clickhouse-gh bot commented Jul 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Algunenano commented Jul 11, 2025 •

edited

Loading

clickhouse-gh bot commented Jul 11, 2025 •

edited

Loading