Skip to content

Join & Aggregation Fine Grained Partition Optimization #6157

@yibin87

Description

@yibin87

Enhancement

This issue tracks Join&Aggregation operator utilizing fine grained partition tech to optimize its implementation.
For Join operator:
The time saved by utilizing fine grained partition tech in Prob side is much fewer than the cost brought by fine grained partition itself, so just consider optimize Build side. Besides, the optimization requires:

  1. Prob side uses the same (or compatible) hash function to choose build side stream, and uses original hash function to do the match work.
  2. The prehash key is the same as the Join hash table's hash key

For Agg operator:
Nothing special.

Related issue: #4631

TiDB

TiFlash

After apply all these optimizations, TPCH_100 3 nodes, 2-tiflash replica, total execution time will reduce about 10%, 80.6 => 72.3:
acbfac11-b14f-4347-b4a6-7cb3fc371e7f

Metadata

Metadata

Assignees

Labels

type/enhancementThe issue or PR belongs to an enhancement.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions