Skip to content

statistics: avoid stats meta full load after table analysis (#57756)#61732

Merged
ti-chi-bot[bot] merged 5 commits intopingcap:release-7.5from
ti-chi-bot:cherry-pick-57756-to-release-7.5
Jun 24, 2025
Merged

statistics: avoid stats meta full load after table analysis (#57756)#61732
ti-chi-bot[bot] merged 5 commits intopingcap:release-7.5from
ti-chi-bot:cherry-pick-57756-to-release-7.5

Conversation

@ti-chi-bot
Copy link
Member

@ti-chi-bot ti-chi-bot commented Jun 13, 2025

This is an automated cherry-pick of #57756

What problem does this PR solve?

Issue Number: close #57631

Problem Summary:

See more at #57631 (comment)

We need to support updating the stats of only a few tables within the update function. This will help reduce the function's duration after the table analysis is completed.

The analysis duration is stable:

mysql> analyze table sbtest111.sbtest2;
Query OK, 0 rows affected, 10 warnings (6.22 sec)

mysql> analyze table sbtest111.sbtest2;
Query OK, 0 rows affected, 10 warnings (6.74 sec)

mysql> analyze table sbtest111.sbtest2;
Query OK, 0 rows affected, 10 warnings (6.12 sec)

What changed and how does it work?

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No need to test
    • I checked and no code files have been changed.

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

@ti-chi-bot ti-chi-bot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. release-note-none Denotes a PR that doesn't merit a release note. sig/planner SIG: Planner size/L Denotes a PR that changes 100-499 lines, ignoring generated files. type/cherry-pick-for-release-7.5 This PR is cherry-picked to release-7.5 from a source PR. labels Jun 13, 2025
@ti-chi-bot ti-chi-bot mentioned this pull request Jun 13, 2025
Merged
13 tasks
@ti-chi-bot
Copy link
Member Author

ti-chi-bot commented Jun 13, 2025

@0xPoe This PR has conflicts, I have hold it.
Please resolve them or ask others to resolve them, then comment /unhold to remove the hold label.

@ti-chi-bot ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jun 13, 2025
@0xPoe
refactor: update stats cache handling to use CacheUpdate struct
40f961f 
This change modifies the UpdateStatsCache method across multiple files to accept a CacheUpdate struct, which encapsulates updated and deleted tables along with options for cache updates. This refactor improves the clarity and flexibility of cache updates, allowing for better management of cache state during statistics updates.

Signed-off-by: 0xPoe <techregister@pm.me>
@0xPoe 0xPoe force-pushed the cherry-pick-57756-to-release-7.5 branch from 1fb4819 to 40f961f Compare June 23, 2025 10:42
@ti-chi-bot ti-chi-bot bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jun 23, 2025
0xPoe added 2 commits June 23, 2025 12:46
@0xPoe
refactor: update benchmark tests to use CacheUpdate struct for stats …
0d60626 
…cache

This update modifies the benchmark tests in bench_test.go to utilize the CacheUpdate struct for the UpdateStatsCache method, enhancing consistency and clarity in cache update operations during performance testing.

Signed-off-by: 0xPoe <techregister@pm.me>
@0xPoe
refactor: replace util with statsutil in UpdateStatsCache call
e22fb02 
This change updates the UpdateStatsCache method in update.go to use the statsutil package instead of the util package, ensuring consistency with recent refactors and improving code clarity.

Signed-off-by: 0xPoe <techregister@pm.me>
@0xPoe 0xPoe requested a review from Copilot June 23, 2025 10:52

This comment was marked as outdated.

Sign in to view

0xPoe added 2 commits June 23, 2025 13:00
@0xPoe
executor: enhance AnalyzeExec to track table and partition IDs
b98d3b8 
This update modifies the AnalyzeExec's Next method to collect and pass table and partition IDs to the statsHandle.Update method, improving the handling of statistics for partitioned tables.

Signed-off-by: 0xPoe <techregister@pm.me>
@0xPoe
test: increase shard count and add new stats cache processing test
3d97d0f 
This commit updates the shard count in the go_test configuration from 13 to 14 for improved test distribution. Additionally, a new test function, TestStatsCacheProcess, is introduced to validate the behavior of the statistics cache during table analysis and updates, ensuring accurate version tracking and cache state management.

Signed-off-by: 0xPoe <techregister@pm.me>
@0xPoe 0xPoe requested a review from Copilot June 23, 2025 11:08
Copilot AI reviewed Jun 23, 2025
View reviewed changes
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR refactors the statistics cache update mechanism to avoid performing a full cache reload after table analysis by updating only the affected tables. Key changes include:

  • Introduction of the CacheUpdate and UpdateOptions types to encapsulate cache updates.
  • New signatures for Update and UpdateStatsCache methods to allow selective updates via table and partition IDs.
  • Updates to various modules (storage, cache, executor, and tests) to support the new update mechanism.

Reviewed Changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
pkg/statistics/handle/util/interfaces.go Defines CacheUpdate and UpdateOptions and updates interface signatures for cache updates.
pkg/statistics/handle/storage/update.go Replaces full cache update calls with the new CacheUpdate mechanism.
pkg/statistics/handle/storage/stats_read_writer.go Uses CacheUpdate in updating stats cache after extended statistics reload.
pkg/statistics/handle/storage/read.go Applies CacheUpdate when updating table statistics after histogram loads.
pkg/statistics/handle/handletest/statstest/stats_test.go Adds tests to verify cache behavior and version management with selective updates.
pkg/statistics/handle/handletest/statstest/BUILD.bazel Minor change adjusting test shard count.
pkg/statistics/handle/handle_hist.go Updates cache update call to use the CacheUpdate struct.
pkg/statistics/handle/handle.go Applies the new CacheUpdate approach when updating partition stats.
pkg/statistics/handle/cache/statscacheinner.go Changes Update function signature to accept the skipMoveForward flag.
pkg/statistics/handle/cache/statscache.go Refactors UpdateStatsCache to use CacheUpdate and conditionally control version updates.
pkg/statistics/handle/cache/bench_test.go Modifies benchmarks to use the new CacheUpdate API.
pkg/executor/analyze.go Enhances update calls by passing table and partition IDs to optimize cache reloads.

0xPoe
0xPoe approved these changes Jun 23, 2025
View reviewed changes
Copy link
Member

@0xPoe 0xPoe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! :shipit:

@ti-chi-bot ti-chi-bot bot added approved needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jun 23, 2025
elsa0520
elsa0520 approved these changes Jun 23, 2025
View reviewed changes
Copy link
Contributor

@elsa0520 elsa0520 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-chi-bot ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jun 23, 2025
@ti-chi-bot
Copy link

ti-chi-bot bot commented Jun 23, 2025

[LGTM Timeline notifier]

Timeline:

  • 2025-06-23 11:14:54.92272638 +0000 UTC m=+702347.645905362: ☑️ agreed by 0xPoe.
  • 2025-06-23 11:21:18.380317187 +0000 UTC m=+702731.103496168: ☑️ agreed by elsa0520.

@ti-chi-bot
Copy link

ti-chi-bot bot commented Jun 23, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 0xPoe, elsa0520, winoros

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@0xPoe
Copy link
Member

0xPoe commented Jun 23, 2025

/unhold

@ti-chi-bot ti-chi-bot bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 23, 2025
@codecov
Copy link

codecov bot commented Jun 23, 2025 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Please upload report for BASE (release-7.5@bfab5eb). Learn more about missing BASE report.

Additional details and impacted files 
@@               Coverage Diff                @@
##             release-7.5     #61732   +/-   ##
================================================
  Coverage               ?   72.2053%           
================================================
  Files                  ?       1417           
  Lines                  ?     414310           
  Branches               ?          0           
================================================
  Hits                   ?     299154           
  Misses                 ?      95148           
  Partials               ?      20008
Flag Coverage Δ
unit 72.2053% <100.0000%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
dumpling 52.9400% <0.0000%> (?)
parser ∅ <0.0000%> (?)
br 53.5305% <0.0000%> (?)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@ti-chi-bot ti-chi-bot bot added cherry-pick-approved Cherry pick PR approved by release team. and removed do-not-merge/cherry-pick-not-approved labels Jun 24, 2025
@ti-chi-bot ti-chi-bot bot merged commit b37cf0e into pingcap:release-7.5 Jun 24, 2025
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@winoros winoros winoros approved these changes

@0xPoe 0xPoe 0xPoe approved these changes

@elsa0520 elsa0520 elsa0520 approved these changes

Assignees

@0xPoe 0xPoe

Labels

approved cherry-pick-approved Cherry pick PR approved by release team. lgtm release-note-none Denotes a PR that doesn't merit a release note. sig/planner SIG: Planner size/L Denotes a PR that changes 100-499 lines, ignoring generated files. type/cherry-pick-for-release-7.5 This PR is cherry-picked to release-7.5 from a source PR.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@ti-chi-bot @0xPoe @winoros @elsa0520