Limit memory usage of exchange sender (#10387)#10417
Limit memory usage of exchange sender (#10387)#10417ti-chi-bot[bot] merged 3 commits intopingcap:release-8.5from
Conversation
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
|
@windtalker This PR has conflicts, I have hold it. |
|
@ti-chi-bot: ## If you want to know how to resolve it, please read the guide in TiDB Dev Guide. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: gengliqi, windtalker The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
[LGTM Timeline notifier]Timeline:
|
|
/hold cancel |
|
/run-all-tests |
|
/test pull-unit-test |
|
/test pull-integration-test |
|
/test pull-unit-test |
|
/test pull-integration-test |
This is an automated cherry-pick of #10387
What problem does this PR solve?
Issue Number: close #10337
Problem Summary:
The root cause is in current implementation, the buffered block number in ExchangeSender can be upto max_threads * max_threads * fine_grain_stream_count, so if max_threads is large, and the size of each row is big, the buffered block can consume very large memory.
This pr limit the memory usage of ExchangeSender by
max(MAX_BATCH_SEND_MIN_LIMIT_MEM_SIZE * max_threads, max_buffered_bytes_in_executor)What is changed and how it works?
Check List
Tests
test in 2-tiflash's cluster, with max_threads = 32 and sort memory limit set to 5G
Where
testhas40000000rows, each row has ~1200 bytes.Before this pr
peak memory of each TiFlash is about 30G
After this pr
peak memory of each TiFlash is about 10G
Side effects
Documentation
Release note