Skip to content

infoschema: refine info cache logic to reduce the impact of DDL on information schema cache#48293

Merged
crazycs520 merged 2 commits intopingcap:tidb-6.5-with-kv-timeout-featurefrom
crazycs520:refine-info-cache-6.5-timeout
Nov 7, 2023
Merged

infoschema: refine info cache logic to reduce the impact of DDL on information schema cache#48293
crazycs520 merged 2 commits intopingcap:tidb-6.5-with-kv-timeout-featurefrom
crazycs520:refine-info-cache-6.5-timeout

Conversation

@crazycs520
Copy link
Contributor

@crazycs520 crazycs520 commented Nov 4, 2023 •
edited
Loading

cherry-pick #48284

What problem does this PR solve?

Issue Number: close #48285

Problem Summary: refine info cache logic to reduce the impact of DDL on information schema cache

What is changed and how it works?

Do same test in #48285, with this PR, the impact of failed to get schema version to stale read QPS is much smaller.

[2023/11/04 09:44:04.794 +08:00] [WARN] [domain.go:196] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=3234]
[2023/11/04 09:44:18.787 +08:00] [WARN] [domain.go:196] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=3280]
[2023/11/04 09:44:20.885 +08:00] [WARN] [domain.go:196] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=3281]
[2023/11/04 09:44:37.502 +08:00] [WARN] [domain.go:196] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=3332]
[2023/11/04 09:44:48.265 +08:00] [WARN] [domain.go:196] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=3363]
[2023/11/04 09:45:10.305 +08:00] [WARN] [domain.go:196] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=3431]
[2023/11/04 09:45:34.291 +08:00] [WARN] [domain.go:196] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=3499]
[2023/11/04 09:45:45.170 +08:00] [WARN] [domain.go:196] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=3531]
[2023/11/04 09:46:12.471 +08:00] [WARN] [domain.go:196] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=3612]
[2023/11/04 09:46:20.786 +08:00] [WARN] [domain.go:196] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=3638]
[2023/11/04 09:46:24.319 +08:00] [WARN] [domain.go:196] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=3648]
[2023/11/04 09:46:26.283 +08:00] [WARN] [domain.go:196] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=3655]
[2023/11/04 09:46:45.023 +08:00] [WARN] [domain.go:196] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=3708]
[2023/11/04 09:46:50.798 +08:00] [WARN] [domain.go:196] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=3728]
[2023/11/04 09:47:12.034 +08:00] [WARN] [domain.go:196] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=3785]
[2023/11/04 09:47:39.571 +08:00] [WARN] [domain.go:196] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=3869]
[2023/11/04 09:47:45.348 +08:00] [WARN] [domain.go:196] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=3888]
[2023/11/04 09:49:10.819 +08:00] [WARN] [domain.go:196] ["failed to get schema version"] [error="There is no Write MVCC info for the schema version"] [version=4136]

The the related metric is following, as you can see, the QPS is much more stable and load snapshot ops is much lower.

image

Before This PR

image

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No need to test
    • I checked and no code files have been changed.

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

@crazycs520
refine info cache
e3c5a4c 
Signed-off-by: crazycs520 <crazycs520@gmail.com>
@crazycs520
update
c7503b6 
Signed-off-by: crazycs520 <crazycs520@gmail.com>
@ti-chi-bot ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Nov 4, 2023
@tiprow
Copy link

tiprow bot commented Nov 4, 2023

Hi @crazycs520. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

crazycs520 added a commit to crazycs520/tidb that referenced this pull request Nov 4, 2023
@crazycs520
infoschema: refine info cache logic to reduce the impact of DDL on in…
06da5f9 
…formation schema cache (pingcap#48293)

Signed-off-by: crazycs520 <crazycs520@gmail.com>
@crazycs520 crazycs520 mentioned this pull request Nov 4, 2023
Closed
cfzjywxk
cfzjywxk approved these changes Nov 4, 2023
View reviewed changes
Copy link
Contributor

@cfzjywxk cfzjywxk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-chi-bot ti-chi-bot bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Nov 4, 2023
ekexium
ekexium approved these changes Nov 7, 2023
View reviewed changes
Copy link
Member

@ekexium ekexium left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can also pick the comments from #48284

@ti-chi-bot ti-chi-bot bot added the lgtm label Nov 7, 2023
@ti-chi-bot
Copy link

ti-chi-bot bot commented Nov 7, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cfzjywxk, ekexium

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot removed the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Nov 7, 2023
@ti-chi-bot
Copy link

ti-chi-bot bot commented Nov 7, 2023

[LGTM Timeline notifier]

Timeline:

  • 2023-11-04 15:13:05.083707203 +0000 UTC m=+3311582.670817475: ☑️ agreed by cfzjywxk.
  • 2023-11-07 04:50:25.111065057 +0000 UTC m=+3533422.698175184: ☑️ agreed by ekexium.

@crazycs520 crazycs520 merged commit 868456d into pingcap:tidb-6.5-with-kv-timeout-feature Nov 7, 2023
guoshouyan pushed a commit to guoshouyan/tidb that referenced this pull request Mar 5, 2024
@Tema @crazycs520
infoschema: refine info cache logic to reduce the impact of DDL on in…
0472bc6 
…formation schema cache pingcap#48293 (pingcap#19)

* refine info cache

Signed-off-by: crazycs520 <crazycs520@gmail.com>

* update

Signed-off-by: crazycs520 <crazycs520@gmail.com>

Signed-off-by: crazycs520 <crazycs520@gmail.com>
Co-authored-by: crazycs520 <crazycs520@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cfzjywxk cfzjywxk cfzjywxk approved these changes

@ekexium ekexium ekexium approved these changes

@zyguan zyguan Awaiting requested review from zyguan

@you06 you06 Awaiting requested review from you06

@MyonKeminta MyonKeminta Awaiting requested review from MyonKeminta

Assignees

No one assigned

Labels

approved lgtm release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@crazycs520 @cfzjywxk @ekexium