in_memory_engine: adjust the default config by glorv · Pull Request #17751 · tikv/tikv

glorv · 2024-10-31T10:53:20Z

What is changed and how it works?

Issue Number: ref #16141, close #17762

What's Changed:

Let in_memory_engine's config`evict-threshold` and `stop-load-threshold`
default value generated from `capacity`.

in_memory_engine config default value:
- evict-threshold: default value set to capacity - min(capacity * 0.1, 200MiB)
- stop-load-threshold: default value set to: evict_threshold - min(capacity * 0.15, region_split_size * 2 + 200MiB)
- set the minimal value of load-evict-interval to 120s as the background task will skip the check if the interval is small than 90s.
Demote some logs in LoadEvictObserver from info to debug as RegionManager will also output these logs.
Change 2 IME load/evcit metrics expression from rate to delta as the rate is too small and we care the actual event number than its rate.

Related changes

PR to update pingcap/docs/pingcap/docs-cn:
Need to cherry-pick to the release branch

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Release note

None

Signed-off-by: glorv <glorvs@163.com>

…t-cfg

overvenus

Rest LGTM

overvenus · 2024-11-01T05:13:51Z

components/in_memory_engine/src/config.rs

 // The maximum interval for GC run is 10 minutes which equals to the minimum
 // value of TiDB GC lifetime.
 const MAX_GC_RUN_INTERVAL: Duration = Duration::from_secs(600);
+// the maximum write kv throughput(20MiB), this is an empiracal value.


Suggested change

// the maximum write kv throughput(20MiB), this is an empiracal value.

// the maximum write kv throughput(20MiB), this is an empirical value.

overvenus · 2024-11-01T05:13:59Z

components/in_memory_engine/src/config.rs

 const MAX_GC_RUN_INTERVAL: Duration = Duration::from_secs(600);
+// the maximum write kv throughput(20MiB), this is an empiracal value.
+const MAX_WRITE_KV_SPEED: u64 = 20 * 1024 * 1024;
+// The maximun duration in seconds we expect IME to release enough memory after


Suggested change

// The maximun duration in seconds we expect IME to release enough memory after

// The maximum duration in seconds we expect IME to release enough memory after

overvenus · 2024-11-01T05:14:20Z

components/in_memory_engine/src/config.rs

+// We use this value to determine the default value of `evict_threshold` based
+// on `capacity`.
+const MAX_RESERVED_DURATION_FOR_WRITE: u64 = 10;
+// Regions' mvcc read amplificatoin statistics is updated every 1min, so we set


Suggested change

// Regions' mvcc read amplificatoin statistics is updated every 1min, so we set

// Regions' mvcc read amplification statistics is updated every 1min, so we set

Signed-off-by: glorv <glorvs@163.com>

…t-cfg

glorv · 2024-11-01T08:21:45Z

/retest

overvenus

Rest LGTM

overvenus · 2024-11-01T09:35:59Z

components/in_memory_engine/src/config.rs

+// the maximum write kv throughput(20MiB), this is an empirical value.
+const MAX_WRITE_KV_SPEED: u64 = 20 * 1024 * 1024;
+// The maximum duration in seconds we expect IME to release enough memory after
+// memory usage reaches `evict_threshold`. This is an empiracal value.


Suggested change

// memory usage reaches `evict_threshold`. This is an empiracal value.

// memory usage reaches `evict_threshold`. This is an empirical value.

ti-chi-bot · 2024-11-01T09:36:28Z

[LGTM Timeline notifier]

Timeline:

2024-11-01 06:31:06.186804684 +0000 UTC m=+590579.025960222: ☑️ agreed by SpadeA-Tang.
2024-11-01 09:36:27.538416544 +0000 UTC m=+601700.377572085: ☑️ agreed by overvenus.

Signed-off-by: glorv <glorvs@163.com>

glorv · 2024-11-03T12:21:46Z

/retest

glorv · 2024-11-04T03:54:30Z

/retest

glorv · 2024-11-04T06:54:13Z

@zhangjinpeng87 PTAL

ti-chi-bot · 2024-11-05T01:28:41Z

@glorv: Your PR was out of date, I have automatically updated it for you.

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

zhangjinpeng87

LGTM

ti-chi-bot · 2024-11-05T03:21:18Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: overvenus, SpadeA-Tang, zhangjinpeng87

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [SpadeA-Tang,overvenus,zhangjinpeng87]
~~src/config/OWNERS~~ [zhangjinpeng87]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2024-11-05T03:25:40Z

In response to a cherrypick label: new pull request created to branch release-8.5: #17770.

ref tikv#16141, close tikv#17762 Let in_memory_engine's config`evict-threshold` and `stop-load-threshold` default value generated from `capacity`. Signed-off-by: glorv <glorvs@163.com>

ref #16141, close #17762 Let in_memory_engine's config`evict-threshold` and `stop-load-threshold` default value generated from `capacity`. Signed-off-by: glorv <glorvs@163.com> Co-authored-by: glorv <glorvs@163.com>

adjust the default config of in_memory_engine

4bb4563

Signed-off-by: glorv <glorvs@163.com>

ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. dco-signoff: yes Indicates the PR's author has signed the dco. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Oct 31, 2024

Merge branch 'master' of https://github.com/tikv/tikv into ime-defaul…

d7c9bf1

…t-cfg

glorv requested review from SpadeA-Tang and overvenus October 31, 2024 10:57

overvenus reviewed Nov 1, 2024

View reviewed changes

glorv added 2 commits November 1, 2024 14:24

fix typos

e7790de

Signed-off-by: glorv <glorvs@163.com>

Merge branch 'master' of https://github.com/tikv/tikv into ime-defaul…

5960234

…t-cfg

glorv requested a review from zhangjinpeng87 November 1, 2024 06:25

SpadeA-Tang approved these changes Nov 1, 2024

View reviewed changes

ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Nov 1, 2024

Merge branch 'master' of https://github.com/tikv/tikv into ime-defaul…

ecb5928

…t-cfg

overvenus approved these changes Nov 1, 2024

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Nov 1, 2024

glorv and others added 4 commits November 1, 2024 17:41

fix typo again

4f7f2e3

Signed-off-by: glorv <glorvs@163.com>

adjust code

2d1898e

Signed-off-by: glorv <glorvs@163.com>

Merge branch 'master' into ime-default-cfg

8ccd1aa

Merge branch 'master' into ime-default-cfg

a5ea08f

glorv added the needs-cherry-pick-release-8.5 Should cherry pick this PR to release-8.5 branch. label Nov 4, 2024

Merge branch 'master' into ime-default-cfg

ac51b2c

ti-chi-bot bot added 2 commits November 4, 2024 08:08

Merge branch 'master' into ime-default-cfg

d1929ad

Merge branch 'master' into ime-default-cfg

4c792cd

zhangjinpeng87 approved these changes Nov 5, 2024

View reviewed changes

ti-chi-bot bot added the approved label Nov 5, 2024

ti-chi-bot bot merged commit 601f34d into tikv:master Nov 5, 2024

ti-chi-bot bot added this to the Pool milestone Nov 5, 2024

ti-chi-bot mentioned this pull request Nov 5, 2024

in_memory_engine: adjust the default config (#17751) #17770

Merged

9 tasks

glorv deleted the ime-default-cfg branch November 5, 2024 03:29

	// the maximum write kv throughput(20MiB), this is an empiracal value.
	// the maximum write kv throughput(20MiB), this is an empirical value.

	// The maximun duration in seconds we expect IME to release enough memory after
	// The maximum duration in seconds we expect IME to release enough memory after

	// Regions' mvcc read amplificatoin statistics is updated every 1min, so we set
	// Regions' mvcc read amplification statistics is updated every 1min, so we set

	// memory usage reaches `evict_threshold`. This is an empiracal value.
	// memory usage reaches `evict_threshold`. This is an empirical value.

Conversation

glorv commented Oct 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is changed and how it works?

Related changes

Check List

Release note

Uh oh!

overvenus left a comment

Choose a reason for hiding this comment

Uh oh!

overvenus Nov 1, 2024

Choose a reason for hiding this comment

Uh oh!

overvenus Nov 1, 2024

Choose a reason for hiding this comment

Uh oh!

overvenus Nov 1, 2024

Choose a reason for hiding this comment

Uh oh!

glorv commented Nov 1, 2024

Uh oh!

overvenus left a comment

Choose a reason for hiding this comment

Uh oh!

overvenus Nov 1, 2024

Choose a reason for hiding this comment

Uh oh!

ti-chi-bot bot commented Nov 1, 2024

[LGTM Timeline notifier]

Uh oh!

glorv commented Nov 3, 2024

Uh oh!

glorv commented Nov 4, 2024

Uh oh!

glorv commented Nov 4, 2024

Uh oh!

ti-chi-bot bot commented Nov 5, 2024

Uh oh!

zhangjinpeng87 left a comment

Choose a reason for hiding this comment

Uh oh!

ti-chi-bot bot commented Nov 5, 2024

Uh oh!

ti-chi-bot commented Nov 5, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

glorv commented Oct 31, 2024 •

edited

Loading