Skip to content

tikv_alloc: Add per thread memory usage#16255

Merged
ti-chi-bot[bot] merged 28 commits intotikv:masterfrom
Connor1996:mem
May 6, 2024
Merged

tikv_alloc: Add per thread memory usage#16255
ti-chi-bot[bot] merged 28 commits intotikv:masterfrom
Connor1996:mem

Conversation

@Connor1996
Copy link
Member

@Connor1996 Connor1996 commented Dec 29, 2023

What is changed and how it works?

Issue Number: Ref #15927

What's Changed:

Add per thread memory usage

Jemalloc has mapped memory stats per arena. We can bind each thread with a dedicated arena, then we have per thread memory usage.

The total memory ~== block cache size + total of per thread footprint
截屏2024-04-07 16 28 25

Related changes

  • PR to update pingcap/docs/pingcap/docs-cn:
  • Need to cherry-pick to the release branch

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Release note

Add per thread memory usage

Signed-off-by: Connor1996 <zbk602423539@gmail.com>
Signed-off-by: Connor1996 <zbk602423539@gmail.com>
Signed-off-by: Connor1996 <zbk602423539@gmail.com>
Signed-off-by: Connor1996 <zbk602423539@gmail.com>
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Dec 29, 2023

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • glorv
  • overvenus

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Details

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Dec 29, 2023
Signed-off-by: Connor1996 <zbk602423539@gmail.com>
let t = thread::Builder::new()
.name(name)
.spawn_wrapper(move || {
tikv_alloc::thread_allocate_exclusive_arena().unwrap();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it mean memory arena can not be shared between threads?
Will it aggravate memory fragment or even OOM?
Can it tell how much memory is held (but not allocated) by a thread?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • By default, the number of arena is four times of CPU cores. So currently, it's already nearly one arena per one thread. So I don't think it would aggravate memory fragment in TiKV case.

  • What you mean by "held" is holding the memory allocated by other thread? Can't as allocator doesn't have the knowledge that the memory ownership is moved to another thread. At least, when it's freed in other threads, the memory usage accounted in the original allocated thread would be decreased.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we consider implementing a configuration switch to disable this feature, as a precautionary measure in case it triggers unforeseen issues?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to the document, " There is a small fixed per-arena overhead, and additionally, arenas manage memory completely independently of each other, which means a small fixed increase in overall memory fragmentation. These overheads are not generally an issue, given the number of arenas normally used. Note that using substantially more arenas than the default is not likely to improve performance, mainly due to reduced cache performance. However, it may make sense to reduce the number of arenas if an application does not make much use of the allocation functions."

To make sure the overhead is small, can you run some read write mixed workload to compare the memory usage with and without this feature? @Connor1996

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are four rounds of sysbench tests, read_only, insert, write_only, read_write case respectively. And there is no noticeable difference in memory usage
07ce2f8c-43c6-4ca9-9d31-1e161785afbc

Signed-off-by: Connor1996 <zbk602423539@gmail.com>
@ti-chi-bot ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Apr 2, 2024
Signed-off-by: Connor1996 <zbk602423539@gmail.com>
Signed-off-by: Connor1996 <zbk602423539@gmail.com>
Signed-off-by: Connor1996 <zbk602423539@gmail.com>
Signed-off-by: Connor1996 <zbk602423539@gmail.com>
Signed-off-by: Connor1996 <zbk602423539@gmail.com>
Signed-off-by: Connor1996 <zbk602423539@gmail.com>
Signed-off-by: Connor1996 <zbk602423539@gmail.com>
Signed-off-by: Connor1996 <zbk602423539@gmail.com>
@ti-chi-bot ti-chi-bot bot removed the status/can-merge Indicates a PR has been approved by a committer. label May 6, 2024
@Connor1996
Copy link
Member Author

/merge

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented May 6, 2024

@Connor1996: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented May 6, 2024

This pull request has been accepted and is ready to merge.

DetailsCommit hash: c4fdfa9

@ti-chi-bot ti-chi-bot bot added the status/can-merge Indicates a PR has been approved by a committer. label May 6, 2024
@ti-chi-bot ti-chi-bot bot merged commit 88099c9 into tikv:master May 6, 2024
@ti-chi-bot ti-chi-bot bot added this to the Pool milestone May 6, 2024
@Connor1996 Connor1996 deleted the mem branch May 6, 2024 06:40
@Connor1996 Connor1996 mentioned this pull request May 7, 2024
16 tasks
@Connor1996
Copy link
Member Author

/cherry-pick release-8.1

ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this pull request May 13, 2024
ref tikv#15927

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot
Copy link
Member

@Connor1996: new pull request created to branch release-8.1: #17005.

Details

In response to this:

/cherry-pick release-8.1

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot bot added a commit that referenced this pull request May 13, 2024
ref #15927

Add per thread memory usage

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Co-authored-by: Connor <zbk602423539@gmail.com>
Co-authored-by: Connor1996 <zbk602423539@gmail.com>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@hhwyt hhwyt added the affects-7.5 This bug affects the 7.5.x(LTS) versions. label Jun 18, 2025
@hhwyt
Copy link
Contributor

hhwyt commented Jun 18, 2025

/cherry-pick release-7.5

ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this pull request Jun 18, 2025
ref tikv#15927

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot
Copy link
Member

@hhwyt: new pull request created to branch release-7.5: #18563.
But this PR has conflicts, please resolve them!

Details

In response to this:

/cherry-pick release-7.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

hhwyt pushed a commit to ti-chi-bot/tikv that referenced this pull request Jun 18, 2025
Signed-off-by: hhwyt <hhwyt1@gmail.com>
hhwyt pushed a commit to hhwyt/tikv that referenced this pull request Jun 19, 2025
Signed-off-by: hhwyt <hhwyt1@gmail.com>
hhwyt pushed a commit to ti-chi-bot/tikv that referenced this pull request Jun 19, 2025
Signed-off-by: hhwyt <hhwyt1@gmail.com>
ekexium pushed a commit to ekexium/tikv that referenced this pull request Jul 11, 2025
Signed-off-by: hhwyt <hhwyt1@gmail.com>
ti-chi-bot bot pushed a commit that referenced this pull request Jul 11, 2025
ref #15927

Add per thread memory usage

Signed-off-by: hhwyt <hhwyt1@gmail.com>

Co-authored-by: Connor <zbk602423539@gmail.com>
bufferflies pushed a commit to bufferflies/tikv that referenced this pull request Sep 19, 2025
…age tikv#16255" into CSE. (tikv#2626)

* [Cherry-pick after rebase] port "tikv_alloc: Add per thread memory usage tikv#16255" into CSE.

Signed-off-by: lucasliang <nkcs_lykx@hotmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

affects-7.5 This bug affects the 7.5.x(LTS) versions. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants