Skip to content

resrc_mtr: add summary mod with a common framework#10973

Closed
mornyx wants to merge 37 commits intotikv:masterfrom
mornyx:resource-metering
Closed

resrc_mtr: add summary mod with a common framework#10973
mornyx wants to merge 37 commits intotikv:masterfrom
mornyx:resource-metering

Conversation

@mornyx
Copy link
Contributor

@mornyx mornyx commented Sep 22, 2021

What problem does this PR solve?

Added support for the collection and reporting of summary data recording the number of scanned keys. In addition, a general resource metering framework is abstracted, and cpu/summary related codes are used as sub-modules to work under the new framework.

What is changed and how it works?

What's Changed:

  • Add a new public abstraction layer, including public models such as Recorder, Reporter and Collector.
  • Support the processing of summary statistics under the public abstraction layer.
  • Migrate the processing of cpu statistics to the public abstraction layer.
  • The only background thread supports collecting multiple resources at the same time.
  • Insert the code to collect scanned keys in the main logic.
  • Add a lot of unit tests.

Related changes

Check List

Tests

  • Unit test
  • Integration test

Side effects

  • Performance regression
    • Consumes more CPU
    • Consumes more MEM

Release note

Support the statistics of scanned keys in resource metering.

crazycs520 and others added 18 commits July 28, 2021 23:38
Signed-off-by: crazycs <chen.two.cs@gmail.com>
Signed-off-by: crazycs <chen.two.cs@gmail.com>
Signed-off-by: crazycs <chen.two.cs@gmail.com>
Signed-off-by: crazycs <chen.two.cs@gmail.com>
Signed-off-by: crazycs <crazycs520@gmail.com>
Signed-off-by: crazycs <chen.two.cs@gmail.com>
Signed-off-by: crazycs <chen.two.cs@gmail.com>
Signed-off-by: crazycs <crazycs520@gmail.com>
Signed-off-by: crazycs <crazycs520@gmail.com>
Signed-off-by: crazycs520 <crazycs520@gmail.com>
fix cargo clippy
adding todos for rw keys

Signed-off-by: lemonhx <lemonhx@lemonhx.tech>
Signed-off-by: lemonhx <lemonhx@lemonhx.tech>
Signed-off-by: mornyx <mornyx.z@gmail.com>
Signed-off-by: mornyx <mornyx.z@gmail.com>
Signed-off-by: mornyx <mornyx.z@gmail.com>
@ti-chi-bot
Copy link
Member

[REVIEW NOTIFICATION]

This pull request has not been approved.

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Details

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Sep 22, 2021
@ti-srebot ti-srebot added the first-time-contributor Indicates that the PR was contributed by an external member and is a first-time contributor. label Sep 22, 2021
@ti-chi-bot ti-chi-bot added the contribution This PR is from a community contributor. label Sep 22, 2021
@ti-chi-bot
Copy link
Member

Welcome @mornyx!

It looks like this is your first PR to tikv/tikv 🎉.

I'm the bot to help you request reviewers, add labels and more, See available commands.

We want to make sure your contribution gets all the attention it needs!



Thank you, and welcome to tikv/tikv. 😃

@ti-chi-bot ti-chi-bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Sep 22, 2021
Signed-off-by: mornyx <mornyx.z@gmail.com>
Signed-off-by: mornyx <mornyx.z@gmail.com>
Signed-off-by: mornyx <mornyx.z@gmail.com>
@ti-chi-bot ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 23, 2021
Signed-off-by: mornyx <mornyx.z@gmail.com>
Signed-off-by: mornyx <mornyx.z@gmail.com>
@ti-chi-bot ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 23, 2021
Signed-off-by: mornyx <mornyx.z@gmail.com>
Signed-off-by: mornyx <mornyx.z@gmail.com>
Signed-off-by: mornyx <mornyx.z@gmail.com>
…re/worker/pd.rs

Signed-off-by: mornyx <mornyx.z@gmail.com>
Signed-off-by: mornyx <mornyx.z@gmail.com>
Signed-off-by: mornyx <mornyx.z@gmail.com>
@mornyx mornyx changed the title resource_metering: add summary module with a general resource metering framework resrc_mtr: add summary mod with a common framework Sep 23, 2021
Signed-off-by: mornyx <mornyx.z@gmail.com>
Comment on lines +3 to +20
/// `Collector` is used to connect [Recorder] and [Reporter].
///
/// The `Recorder` is mainly responsible for collecting data, and it is
/// only responsible for passing the collected data to the `Collector`.
/// The `Recorder` does not know anything about "scheduling" or "uploading".
///
/// Typically, constructing a `Collector` instance requires passing in a
/// [Scheduler], The `Collector` will send the data passed by the recorder
/// to the `Scheduler` for processing.
///
/// `Reporter` implements [Runnable] and [RunnableWithTimer], aggregates the
/// data sent by the `Collector` internally, and reports it regularly through RPC.
///
/// [Recorder]: crate::recorder::Recorder
/// [Reporter]: crate::reporter::Reporter
/// [Scheduler]: tikv_util::worker::Scheduler
/// [Runnable]: tikv_util::worker::Runnable
/// [RunnableWithTimer]: tikv_util::worker::RunnableWithTimer
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Signed-off-by: mornyx <mornyx.z@gmail.com>
Signed-off-by: mornyx <mornyx.z@gmail.com>
Signed-off-by: mornyx <mornyx.z@gmail.com>
Signed-off-by: mornyx <mornyx.z@gmail.com>
Signed-off-by: mornyx <mornyx.z@gmail.com>
Signed-off-by: mornyx <mornyx.z@gmail.com>
Signed-off-by: mornyx <mornyx.z@gmail.com>
@ti-chi-bot ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 9, 2021
@ti-chi-bot
Copy link
Member

@mornyx: PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@mornyx
Copy link
Contributor Author

mornyx commented Oct 18, 2021

This PR was split into two: 10998, 11085

@mornyx mornyx closed this Oct 18, 2021
@mornyx mornyx deleted the resource-metering branch December 14, 2021 17:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

contribution This PR is from a community contributor. first-time-contributor Indicates that the PR was contributed by an external member and is a first-time contributor. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants