Skip to content

resource_metering: Aggregate RawRecords before picking topN records#18834

Merged
ti-chi-bot[bot] merged 28 commits intotikv:masterfrom
yibin87:add_aggregator_for_topsql
Aug 22, 2025
Merged

resource_metering: Aggregate RawRecords before picking topN records#18834
ti-chi-bot[bot] merged 28 commits intotikv:masterfrom
yibin87:add_aggregator_for_topsql

Conversation

@yibin87
Copy link
Contributor

@yibin87 yibin87 commented Aug 14, 2025 •
edited
Loading

What is changed and how it works?

Issue Number: Close #18844

What's Changed:

Add an aggregator to aggregate RawRecords by the extra tag before picking topN records.

Related changes

  • PR to update pingcap/docs/pingcap/docs-cn:
  • Need to cherry-pick to the release branch

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
    Test the TPCH10 benchmark, and check the other time drops from 38% to 0.0033%.
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Release note

It improves the accuracy of data collected for topsql.

@yibin87
Quick demo for adding agg
7f7526d 
Signed-off-by: yibin87 <huyibin@pingcap.com>
@ti-chi-bot ti-chi-bot bot added do-not-merge/needs-linked-issue dco-signoff: yes Indicates the PR's author has signed the dco. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. contribution This PR is from a community contributor. labels Aug 14, 2025
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Aug 14, 2025

Hi @yibin87. Thanks for your PR.

I'm waiting for a tikv member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@ti-chi-bot ti-chi-bot bot added needs-ok-to-test Indicates a PR created by contributors and need ORG member send '/ok-to-test' to start testing. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Aug 14, 2025
@yibin87 yibin87 changed the title [DNM] Quick demo for adding agg resource_metering: Aggregate RawRecords before pick topN records Aug 19, 2025
@yibin87 yibin87 changed the title resource_metering: Aggregate RawRecords before pick topN records [WIP]resource_metering: Aggregate RawRecords before pick topN records Aug 19, 2025
@ti-chi-bot ti-chi-bot bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 19, 2025
@yibin87
Little refact
4a7e57e 
Signed-off-by: yibin87 <huyibin@pingcap.com>
@ti-chi-bot ti-chi-bot bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Aug 20, 2025
@yibin87
Little refact
68f67eb 
Signed-off-by: yibin87 <huyibin@pingcap.com>
@ti-chi-bot ti-chi-bot bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Aug 20, 2025
yibin87 added 2 commits August 20, 2025 10:09
@yibin87
Update related tests
d45a3b1 
Signed-off-by: yibin87 <huyibin@pingcap.com>
@yibin87
Little update
a6b663c 
Signed-off-by: yibin87 <huyibin@pingcap.com>
@yibin87 yibin87 changed the title [WIP]resource_metering: Aggregate RawRecords before pick topN records resource_metering: Aggregate RawRecords before pick topN records Aug 20, 2025
@ti-chi-bot ti-chi-bot bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 20, 2025
yibin87 added 4 commits August 20, 2025 10:46
@yibin87
Fix test compilation
e3e9083 
Signed-off-by: yibin87 <huyibin@pingcap.com>
@yibin87
Little update
6a23dbf 
Signed-off-by: yibin87 <huyibin@pingcap.com>
@yibin87
Little update
ba47e0c 
Signed-off-by: yibin87 <huyibin@pingcap.com>
@yibin87
Fix test issues
9209312 
Signed-off-by: yibin87 <huyibin@pingcap.com>
nolouch
nolouch reviewed Aug 20, 2025
View reviewed changes
yibin87 added 3 commits August 20, 2025 14:07
@yibin87
Address comments
3cc1d5f 
Signed-off-by: yibin87 <huyibin@pingcap.com>
@yibin87
Little refact
ef4f7d3 
Signed-off-by: yibin87 <huyibin@pingcap.com>
@yibin87
Add ut for aggregator
9ca654c 
Signed-off-by: yibin87 <huyibin@pingcap.com>
@yibin87
Replace into with arc new
69bd35b 
Signed-off-by: yibin87 <huyibin@pingcap.com>
@yibin87
Copy link
Contributor Author

yibin87 commented Aug 21, 2025

/cc @glorv

@ti-chi-bot ti-chi-bot bot requested a review from glorv August 21, 2025 08:29
nolouch
nolouch reviewed Aug 21, 2025
View reviewed changes
Copy link
Contributor

@nolouch nolouch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rest lgtm

components/resource_metering/src/reporter/mod.rs Outdated
};

thread_local! {
static STATIC_BUF: Cell<Vec<u32>> = const {Cell::new(vec![])};
Copy link
Contributor

@nolouch nolouch Aug 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it useless?

Copy link
Contributor Author

@yibin87 yibin87 Aug 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, removed

@nolouch
Copy link
Contributor

nolouch commented Aug 21, 2025

/ok-to-test

@ti-chi-bot ti-chi-bot bot added ok-to-test Indicates a PR is ready to be tested. and removed needs-ok-to-test Indicates a PR created by contributors and need ORG member send '/ok-to-test' to start testing. labels Aug 21, 2025
@yibin87
Address comments to remove useless code
691b056 
Signed-off-by: yibin <huyibin@pingcap.com>
@yibin87 yibin87 requested a review from nolouch August 22, 2025 01:51
yibin87 added 4 commits August 22, 2025 10:19
@yibin87
Fix lint issue
e61e86b 
Signed-off-by: yibin <huyibin@pingcap.com>
@yibin87
Fix left lint issues
8a6fb20 
Signed-off-by: yibin <huyibin@pingcap.com>
@yibin87
Little update
0707041 
Signed-off-by: yibin <huyibin@pingcap.com>
@yibin87
Fix ut incorrect result
56bcada 
Signed-off-by: yibin <huyibin@pingcap.com>
@yibin87
Copy link
Contributor Author

yibin87 commented Aug 22, 2025

/test pull-unit-test

glorv
glorv approved these changes Aug 22, 2025
View reviewed changes
Copy link
Contributor

@glorv glorv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-chi-bot ti-chi-bot bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Aug 22, 2025
nolouch
nolouch approved these changes Aug 22, 2025
View reviewed changes
Copy link
Contributor

@nolouch nolouch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@ti-chi-bot ti-chi-bot bot added the lgtm label Aug 22, 2025
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Aug 22, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: glorv, nolouch

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot removed the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Aug 22, 2025
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Aug 22, 2025

[LGTM Timeline notifier]

Timeline:

  • 2025-08-22 07:24:24.505996036 +0000 UTC m=+597472.449171552: ☑️ agreed by glorv.
  • 2025-08-22 18:26:18.341393054 +0000 UTC m=+637186.284568560: ☑️ agreed by nolouch.

@ti-chi-bot ti-chi-bot bot merged commit 0d3302b into tikv:master Aug 22, 2025
5 checks passed
@ti-chi-bot ti-chi-bot bot added this to the Pool milestone Aug 22, 2025
@yibin87
Copy link
Contributor Author

yibin87 commented Aug 25, 2025

/cherry-pick release-8.5

ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this pull request Aug 25, 2025
@yibin87 @ti-chi-bot
This is an automated cherry-pick of tikv#18834
5edae20 
close tikv#18844

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this pull request Aug 25, 2025
Merged
9 tasks
@ti-chi-bot
Copy link
Member

ti-chi-bot commented Aug 25, 2025

@yibin87: new pull request created to branch release-8.5: #18876.
But this PR has conflicts, please resolve them!

Details

In response to this:

/cherry-pick release-8.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

3AceShowHand pushed a commit to 3AceShowHand/tikv that referenced this pull request Oct 13, 2025
@yibin87 @3AceShowHand
resource_metering: Aggregate RawRecords before picking topN records (t…
baa28a9 
…ikv#18834)

close tikv#18844

Add an aggregator to aggregate RawRecords by the extra tag before picking topN records.

Signed-off-by: yibin <huyibin@pingcap.com>
Signed-off-by: 3AceShowHand <jinl1037@hotmail.com>
ti-chi-bot bot pushed a commit that referenced this pull request Nov 11, 2025
@ti-chi-bot @yibin87
resource_metering: Aggregate RawRecords before picking topN records (#…
75f416e 
…18834) (#18876)

close #18844

Add an aggregator to aggregate RawRecords by the extra tag before picking topN records.

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
Signed-off-by: yibin87 <huyibin@pingcap.com>

Co-authored-by: yibin <huyibin@pingcap.com>
Co-authored-by: yibin87 <huyibin@pingcap.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@glorv glorv glorv approved these changes

@breezewish breezewish Awaiting requested review from breezewish

+1 more reviewer

@nolouch nolouch nolouch approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved contribution This PR is from a community contributor. dco-signoff: yes Indicates the PR's author has signed the dco. lgtm ok-to-test Indicates a PR is ready to be tested. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Milestone

Pool

Development

Successfully merging this pull request may close these issues.

Aggregate RawRecords before picking topN records

4 participants

@yibin87 @nolouch @ti-chi-bot @glorv