statistics: avoid stats meta full load after table analysis by 0xPoe · Pull Request #57756 · pingcap/tidb

0xPoe · 2024-11-27T05:53:31Z

What problem does this PR solve?

Issue Number: close #57631

Problem Summary:

We need to support updating the stats of only a few tables within the update function. This will help reduce the function's duration after the table analysis is completed.

The analysis duration is stable:

mysql> analyze table sbtest111.sbtest2;
Query OK, 0 rows affected, 10 warnings (6.22 sec)

mysql> analyze table sbtest111.sbtest2;
Query OK, 0 rows affected, 10 warnings (6.74 sec)

mysql> analyze table sbtest111.sbtest2;
Query OK, 0 rows affected, 10 warnings (6.12 sec)

What changed and how does it work?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

0xPoe · 2024-11-27T06:13:38Z

/retest

codecov · 2024-11-27T06:21:03Z

Codecov Report

Attention: Patch coverage is 95.65217% with 3 lines in your changes missing coverage. Please review.

Project coverage is 73.8288%. Comparing base (8fe0618) to head (3808148).
Report is 1158 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #57756        +/-   ##
================================================
+ Coverage   73.1430%   73.8288%   +0.6858%     
================================================
  Files          1671       1701        +30     
  Lines        460633     468749      +8116     
================================================
+ Hits         336921     346072      +9151     
+ Misses       103006     101245      -1761     
- Partials      20706      21432       +726

Flag	Coverage Δ
integration	`46.1402% <72.4637%> (?)`
unit	`72.5347% <95.6521%> (+0.2543%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`74.9758% <ø> (+2.4522%)`	⬆️
parser	`∅ <ø> (∅)`
br	`48.0234% <ø> (-0.1131%)`	⬇️

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

0xPoe

🔢 Self-check (PR reviewed by myself and ready for feedback.)

0xPoe · 2024-11-27T06:42:45Z

pkg/executor/analyze.go

 	if len(tasks) == 0 {
 		return nil
 	}
+	tableAndPartitionIDs := make([]int64, 0, len(tasks))


Maybe we can get this information from the previous function, but I’d prefer not to introduce extra refactoring in this PR since we need to cherry-pick it to 8.5.

Signed-off-by: Rustin170506 <tech@rustin.me>

0xPoe · 2024-11-29T05:56:45Z

/retest

0xPoe

🔢 Self-check (PR reviewed by myself and ready for feedback.)

0xPoe · 2024-11-29T08:13:01Z

/retest

time-and-fate

Others LGTM.

pkg/statistics/handle/storage/read.go

Signed-off-by: Rustin170506 <tech@rustin.me>

time-and-fate

/hold

pkg/statistics/handle/types/interfaces.go

pkg/statistics/handle/storage/update.go

ti-chi-bot · 2024-12-02T14:03:52Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: time-and-fate, winoros

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [time-and-fate,winoros]
~~pkg/statistics/OWNERS~~ [time-and-fate,winoros]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2024-12-02T14:03:56Z

[LGTM Timeline notifier]

Timeline:

2024-12-02 06:53:40.061517703 +0000 UTC m=+1051407.681172220: ☑️ agreed by winoros.
2024-12-02 14:03:54.941472793 +0000 UTC m=+1077222.561127308: ☑️ agreed by time-and-fate.

Signed-off-by: Rustin170506 <tech@rustin.me>

0xPoe

🔢 Self-check (PR reviewed by myself and ready for feedback.)

0xPoe · 2024-12-03T05:11:16Z

/unhold

ti-chi-bot · 2024-12-03T05:56:02Z

In response to a cherrypick label: new pull request created to branch release-8.5: #57911.

…57911) close #57631

* skip TestIndexJoin31494 * fix * global sort: add boundaries to split keys when generating plan (pingcap#58323) (pingcap#58356) * statistics: get right max table id when to init stats (pingcap#58280) (pingcap#58298) * executor: Fix the parse problematic slow log panic issue due to empty … * statstics: trigger evict by the timer (pingcap#58027) (pingcap#58268) * br: make table existence check unified on different br client (pingcap#58211) (pingcap#58262) * log backup: use global checkpoint ts as source of truth (pingcap#58135) (pingcap#58265) * executor: skip execution when build query for VIEW in I_S (pingcap#58203) (pingcap#58236) * statistics: copy stats when to update it for avoiding data race (pingcap#5810… * domain,infoschema: make infoschema activity block GC safepoint advanci… * planner: handle panic when loading bindings at startup (pingcap#58017) (pingcap#58035) * statistics: right deal with error for reading stats from storage (pingcap#58… * statistics: lite init used wrong value to build table stats ver (pingcap#5802… * lightning, ddl: set TS to engineMeta after ResetEngineSkipAllocTS (pingcap#5… * *: avoid unlock of unlocked mutex panic on TableDeltaMap (pingcap#57799) (pingcap#57997) * ddl: handle context done after sending DDL jobs (pingcap#57945) (pingcap#57989) * *: activate txn for query on infoschema tables (pingcap#57937) (pingcap#57951) * lightning: add PK to internal tables (pingcap#57480) (pingcap#57932) * statistics: correct behavior of non-lite InitStats and stats sync load… * statistics: avoid stats meta full load after table analysis (pingcap#57756) (pingcap#57911) * dumpling: use I_S to get table list for TiDB and add database to WHERE… * br: fix insert gc failed due to slow schema reload (pingcap#57742) (pingcap#57907) * statistics: do not record historical stats meta if the table is locked… * metrics: remove the filled colors (pingcap#57838) (pingcap#57866) * planner: use TableInfo.DBID to locate schema (pingcap#57785) (pingcap#57870) * *: support cancel query like 'select * from information_schema.tables'… * session: make `TxnInfo()` return even if process info is empty (pingcap#57044) (pingcap#57161) * ddl: Fixed partitioning a non-partitioned table with placement rules (… * *: Reorg partition fix delete ranges and handling non-clustered tables… * executor: fix query infoschema.tables table_schema/table_name with fil… * ddl: check context done in isReorgRunnable function (pingcap#57813) (pingcap#57820) * ddl: fix ExistsTableRow and add tests for skip reorg checks (pingcap#57778) (pingcap#57801) * *: Fix for TRUNCATE PARTITION and Global Index (pingcap#57724) * br: prompt k8s.io/api version (pingcap#57791) (pingcap#57802) * statistics: fix some problem related to stats async load (pingcap#57723) (pingcap#57775) * expression: fix wrong calculation order of `radians` (pingcap#57672) (pingcap#57688) * statistics: rightly deal with timout when to send sync load (pingcap#57712) (pingcap#57751) * ddl: `tidb_scatter_region` variable supports setting value in both upp… * planner: fix that vector index output empty result when pk is non-int … * ddl: dynamically adjusting the max write speed of reorganization job (… * executor: fix hang in hash agg when exceeding memory limit leads to pa… * statistics: use infoschema api to get table info (pingcap#57574) (pingcap#57614) * planner: Use realtimeRowCount when all topN collected (pingcap#56848) (pingcap#57689) * statistics: handle deleted tables correctly in the PQ (pingcap#57649) (pingcap#57674) * backup: reset timeout on store level (pingcap#55526) (pingcap#57667) * planner/core: fix a wrong privilege check for CTE & UPDATE statement (…

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2025-06-13T12:54:29Z

In response to a cherrypick label: new pull request created to branch release-7.5: #61732.
But this PR has conflicts, please resolve them!

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2025-06-23T14:56:12Z

In response to a cherrypick label: new pull request created to branch release-8.1: #61944.
But this PR has conflicts, please resolve them!

…61732) close #57631

ti-chi-bot bot added do-not-merge/needs-triage-completed release-note-none Denotes a PR that doesn't merit a release note. sig/planner SIG: Planner size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed do-not-merge/needs-triage-completed labels Nov 27, 2024

0xPoe commented Nov 27, 2024

View reviewed changes

0xPoe requested review from time-and-fate and winoros November 27, 2024 07:00

0xPoe added 4 commits November 28, 2024 11:35

statistics: avoid stats meta full load after table analysis

e82e4b4

Signed-off-by: Rustin170506 <tech@rustin.me>

fix: make lint happy

e18ec37

Signed-off-by: Rustin170506 <tech@rustin.me>

test: add an unit test

8d6e156

Signed-off-by: Rustin170506 <tech@rustin.me>

docs: add comments

519a0d6

Signed-off-by: Rustin170506 <tech@rustin.me>

0xPoe force-pushed the rustin-patch-analysis-full-load branch from 675167d to 519a0d6 Compare November 28, 2024 03:37

0xPoe added 2 commits November 28, 2024 13:34

fix: bump to 10000

0376db2

Signed-off-by: Rustin170506 <tech@rustin.me>

debug: print rows

8576b1e

Signed-off-by: Rustin170506 <tech@rustin.me>

0xPoe force-pushed the rustin-patch-analysis-full-load branch from fa1f501 to 8576b1e Compare November 28, 2024 11:30

0xPoe added 2 commits November 28, 2024 20:40

debug: print delta number

dd2c341

Signed-off-by: Rustin170506 <tech@rustin.me>

fix: remove logs

238a1c4

Signed-off-by: Rustin170506 <tech@rustin.me>

0xPoe commented Nov 29, 2024

View reviewed changes

winoros approved these changes Dec 2, 2024

View reviewed changes

ti-chi-bot bot added approved needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Dec 2, 2024

time-and-fate reviewed Dec 2, 2024

View reviewed changes

pkg/statistics/handle/storage/read.go Outdated Show resolved Hide resolved

fix: remove useless change

6febcf8

Signed-off-by: Rustin170506 <tech@rustin.me>

0xPoe requested a review from time-and-fate December 2, 2024 13:37

time-and-fate approved these changes Dec 2, 2024

View reviewed changes

pkg/statistics/handle/types/interfaces.go Outdated Show resolved Hide resolved

pkg/statistics/handle/storage/update.go Outdated Show resolved Hide resolved

ti-chi-bot bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 2, 2024

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Dec 2, 2024

ti-chi-bot bot added the needs-cherry-pick-release-8.5 Should cherry pick this PR to release-8.5 branch. label Dec 3, 2024

fix: remove unnessary code

b50b414

Signed-off-by: Rustin170506 <tech@rustin.me>

0xPoe commented Dec 3, 2024

View reviewed changes

Merge branch 'master' into rustin-patch-analysis-full-load

3808148

ti-chi-bot bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 3, 2024

ti-chi-bot bot merged commit f585f5d into pingcap:master Dec 3, 2024

ti-chi-bot mentioned this pull request Dec 3, 2024

statistics: avoid stats meta full load after table analysis (#57756) #57911

Merged

13 tasks

ti-chi-bot bot pushed a commit that referenced this pull request Dec 3, 2024

statistics: avoid stats meta full load after table analysis (#57756) (#…

6087f99

…57911) close #57631

0xPoe mentioned this pull request Dec 19, 2024

nightly : The ANALYZE TABLE command does not return a result, but it is actually successful #54475

Closed

0xPoe added the needs-cherry-pick-release-7.5 Should cherry pick this PR to release-7.5 branch. label Jun 13, 2025

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Jun 13, 2025

This is an automated cherry-pick of pingcap#57756

1fb4819

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Jun 13, 2025

statistics: avoid stats meta full load after table analysis (#57756) #61732

Merged

13 tasks

ti-chi-bot bot added the needs-cherry-pick-release-8.1 Should cherry pick this PR to release-8.1 branch. label Jun 23, 2025

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Jun 23, 2025

This is an automated cherry-pick of pingcap#57756

6752f80

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Jun 23, 2025

statistics: avoid stats meta full load after table analysis (#57756) #61944

Open

13 tasks

ti-chi-bot bot pushed a commit that referenced this pull request Jun 24, 2025

statistics: avoid stats meta full load after table analysis (#57756) (#…

b37cf0e

…61732) close #57631

Conversation

0xPoe commented Nov 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

Uh oh!

0xPoe commented Nov 27, 2024

Uh oh!

codecov bot commented Nov 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

0xPoe left a comment

Choose a reason for hiding this comment

Uh oh!

0xPoe Nov 27, 2024

Choose a reason for hiding this comment

Uh oh!

0xPoe commented Nov 29, 2024

Uh oh!

0xPoe left a comment

Choose a reason for hiding this comment

Uh oh!

0xPoe commented Nov 29, 2024

Uh oh!

time-and-fate left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

time-and-fate left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ti-chi-bot bot commented Dec 2, 2024

Uh oh!

ti-chi-bot bot commented Dec 2, 2024

[LGTM Timeline notifier]

Uh oh!

0xPoe left a comment

Choose a reason for hiding this comment

Uh oh!

0xPoe commented Dec 3, 2024

Uh oh!

ti-chi-bot commented Dec 3, 2024

Uh oh!

ti-chi-bot commented Jun 13, 2025

Uh oh!

ti-chi-bot commented Jun 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

0xPoe commented Nov 27, 2024 •

edited

Loading

codecov bot commented Nov 27, 2024 •

edited

Loading