*: support columnar inverted index by Lloyd-Pottiger · Pull Request #59571 · pingcap/tidb

Lloyd-Pottiger · 2025-02-17T03:09:35Z

What problem does this PR solve?

Issue Number: close #59880

Problem Summary:

What changed and how does it work?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

tiprow · 2025-02-17T03:09:55Z

Hi @Lloyd-Pottiger. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

wuhuizuo · 2025-02-17T03:47:07Z

/test ?

ti-chi-bot · 2025-02-17T03:47:10Z

@wuhuizuo: The following commands are available to trigger required jobs:

/test build

/test check-dev

/test check-dev2

/test mysql-test

/test pull-br-integration-test

/test pull-integration-ddl-test

/test pull-integration-e2e-test

/test pull-lightning-integration-test

/test pull-mysql-client-test

/test pull-unit-test-ddlv1

/test unit-test

The following commands are available to trigger optional jobs:

/test pingcap/tidb/canary_ghpr_unit_test

/test pull-common-test

/test pull-e2e-test

/test pull-integration-common-test

/test pull-integration-copr-test

/test pull-integration-jdbc-test

/test pull-integration-mysql-test

/test pull-integration-nodejs-test

/test pull-integration-python-orm-test

/test pull-scan-deps

/test pull-sqllogic-test

/test pull-tiflash-test

Use /test all to run the following jobs that were automatically triggered:

pingcap/tidb/ghpr_build

pingcap/tidb/ghpr_check

pingcap/tidb/ghpr_check2

pingcap/tidb/ghpr_mysql_test

pingcap/tidb/ghpr_unit_test

pingcap/tidb/pull_integration_ddl_test

pingcap/tidb/pull_integration_e2e_test

pingcap/tidb/pull_mysql_client_test

Details

In response to this:

/test ?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tiprow · 2025-02-17T03:47:30Z

@wuhuizuo: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/test ?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tiprow · 2025-02-17T03:47:32Z

@ti-chi-bot[bot]: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

@wuhuizuo: The following commands are available to trigger required jobs:
/test build
/test check-dev
/test check-dev2
/test mysql-test
/test pull-br-integration-test
/test pull-integration-ddl-test
/test pull-integration-e2e-test
/test pull-lightning-integration-test
/test pull-mysql-client-test
/test pull-unit-test-ddlv1
/test unit-test
The following commands are available to trigger optional jobs:
/test pingcap/tidb/canary_ghpr_unit_test
/test pull-common-test
/test pull-e2e-test
/test pull-integration-common-test
/test pull-integration-copr-test
/test pull-integration-jdbc-test
/test pull-integration-mysql-test
/test pull-integration-nodejs-test
/test pull-integration-python-orm-test
/test pull-scan-deps
/test pull-sqllogic-test
/test pull-tiflash-test
Use /test all to run the following jobs that were automatically triggered:
pingcap/tidb/ghpr_build
pingcap/tidb/ghpr_check
pingcap/tidb/ghpr_check2
pingcap/tidb/ghpr_mysql_test
pingcap/tidb/ghpr_unit_test
pingcap/tidb/pull_integration_ddl_test
pingcap/tidb/pull_integration_e2e_test
pingcap/tidb/pull_mysql_client_test
In response to this:

/test ?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

codecov · 2025-03-21T03:48:19Z

Codecov Report

Attention: Patch coverage is 67.29560% with 52 lines in your changes missing coverage. Please review.

Project coverage is 75.3838%. Comparing base (7232aea) to head (51c75c4).
Report is 44 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #59571        +/-   ##
================================================
+ Coverage   73.0943%   75.3838%   +2.2895%     
================================================
  Files          1710       1760        +50     
  Lines        472729     493091     +20362     
================================================
+ Hits         345538     371711     +26173     
+ Misses       105887      98738      -7149     
- Partials      21304      22642      +1338

Flag	Coverage Δ
integration	`49.4612% <20.7547%> (?)`
unit	`72.7446% <65.4088%> (+0.4550%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.6553% <ø> (-0.0357%)`	⬇️
parser	`∅ <ø> (∅)`
br	`62.6295% <ø> (+15.4121%)`	⬆️

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

pkg/ddl/create_table.go

Co-authored-by: tangenta <tangenta@126.com>

Signed-off-by: Lloyd-Pottiger <yan1579196623@gmail.com>

pkg/meta/model/index.go

tangenta · 2025-04-08T09:27:20Z

pkg/types/etc.go

+func IsTypeStoredAsInteger(tp byte) bool {
+	switch tp {
+	case mysql.TypeTiny, mysql.TypeShort, mysql.TypeInt24, mysql.TypeLong, mysql.TypeLonglong:
+		return true
+	case mysql.TypeYear:
+		return true
+	// Enum and Set are stored as integer type but they can not be pushed down to TiFlash
+	// case mysql.TypeEnum, mysql.TypeSet:
+	// 	return true
+	case mysql.TypeDatetime, mysql.TypeDate, mysql.TypeTimestamp, mysql.TypeDuration:
+		return true
+	}
+	return false


Why not compact these cases into one branch?

It is more clear, Integer type and date type.

tangenta

Rest LGTM

ti-chi-bot · 2025-04-08T09:28:54Z

[LGTM Timeline notifier]

Timeline:

2025-04-02 09:37:50.718137428 +0000 UTC m=+1644364.402373524: ☑️ agreed by zimulala.
2025-04-08 09:28:53.432627224 +0000 UTC m=+2162227.116863320: ☑️ agreed by tangenta.

Co-authored-by: tangenta <tangenta@126.com>

winoros · 2025-04-08T18:42:47Z

pkg/planner/core/find_best_task.go

+			if index.State == model.StatePublic && index.InvertedInfo != nil {
+				ts.UsedColumnarIndexes = append(ts.UsedColumnarIndexes, &tipb.ColumnarIndexInfo{
+					IndexType: tipb.ColumnarIndexType_TypeInverted,
+					Index: &tipb.ColumnarIndexInfo_InvertedQueryInfo{
+						InvertedQueryInfo: &tipb.InvertedQueryInfo{
+							IndexId:  index.ID,
+							ColumnId: index.InvertedInfo.ColumnID,
+						},
+					},
+				})
+			}


do we need to check whether the index is used or not?
if the numeric column a has inverted index while this SQL don't have predicates on it, do we need to add that index here?

To simplify, just add all inverted index info now since we all implement a better algorithm soon.

ti-chi-bot · 2025-04-09T09:38:20Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tangenta, winoros, zimulala

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [tangenta,winoros,zimulala]
~~pkg/ddl/OWNERS~~ [tangenta]
~~pkg/meta/OWNERS~~ [tangenta]
~~pkg/parser/OWNERS~~ [tangenta]
~~pkg/planner/OWNERS~~ [winoros]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Lloyd-Pottiger · 2025-04-09T12:22:38Z

/retest

tiprow · 2025-04-09T12:23:01Z

@Lloyd-Pottiger: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Lloyd-Pottiger · 2025-04-10T02:01:56Z

/retest

tiprow · 2025-04-10T02:02:19Z

@Lloyd-Pottiger: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Lloyd-Pottiger · 2025-04-10T09:15:03Z

/retest

tiprow · 2025-04-10T09:15:26Z

@Lloyd-Pottiger: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Lloyd-Pottiger · 2025-04-10T11:59:48Z

/retest

tiprow · 2025-04-10T12:00:10Z

@Lloyd-Pottiger: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

close pingcap#59880

ti-chi-bot bot added do-not-merge/needs-linked-issue do-not-merge/needs-tests-checked release-note-none Denotes a PR that doesn't merit a release note. sig/planner SIG: Planner labels Feb 17, 2025

Lloyd-Pottiger marked this pull request as draft February 17, 2025 03:09

ti-chi-bot bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Feb 17, 2025

Lloyd-Pottiger force-pushed the support-inverted-index branch from b0d47bb to 0900788 Compare February 20, 2025 03:42

ti-chi-bot bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Feb 20, 2025

Lloyd-Pottiger force-pushed the support-inverted-index branch from 0900788 to 3c4e173 Compare February 25, 2025 06:47

Lloyd-Pottiger force-pushed the support-inverted-index branch 4 times, most recently from 5f00d4a to 9ac9cd2 Compare March 17, 2025 06:59

Lloyd-Pottiger force-pushed the support-inverted-index branch from 7541ff3 to 4a21b72 Compare March 21, 2025 03:27

ti-chi-bot bot removed the do-not-merge/needs-linked-issue label Mar 21, 2025

Lloyd-Pottiger changed the title ~~DNM: test~~ *: support columnar inverted index Mar 21, 2025

Lloyd-Pottiger marked this pull request as ready for review March 21, 2025 03:30

ti-chi-bot bot removed do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. do-not-merge/needs-tests-checked labels Mar 21, 2025

breezewish reviewed Mar 21, 2025

View reviewed changes

pkg/ddl/create_table.go Outdated Show resolved Hide resolved

breezewish reviewed Mar 21, 2025

View reviewed changes

pkg/ddl/create_table.go Outdated Show resolved Hide resolved

Lloyd-Pottiger force-pushed the support-inverted-index branch from 5bb5cc7 to 6eb5e41 Compare March 24, 2025 02:55

Lloyd-Pottiger and others added 4 commits April 3, 2025 11:20

Update pkg/ddl/executor.go

0182ae8

Co-authored-by: tangenta <tangenta@126.com>

address comments

dd246c3

Signed-off-by: Lloyd-Pottiger <yan1579196623@gmail.com>

fix

2f0b6ca

Signed-off-by: Lloyd-Pottiger <yan1579196623@gmail.com>

fix

6822cf8

Signed-off-by: Lloyd-Pottiger <yan1579196623@gmail.com>

tangenta reviewed Apr 8, 2025

View reviewed changes

pkg/meta/model/index.go Outdated Show resolved Hide resolved

tangenta reviewed Apr 8, 2025

View reviewed changes

tangenta approved these changes Apr 8, 2025

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Apr 8, 2025

Update pkg/meta/model/index.go

51c75c4

Co-authored-by: tangenta <tangenta@126.com>

winoros reviewed Apr 8, 2025

View reviewed changes

winoros approved these changes Apr 9, 2025

View reviewed changes

ti-chi-bot bot added the approved label Apr 9, 2025

ti-chi-bot bot merged commit 96e00d1 into pingcap:master Apr 10, 2025
25 checks passed

Lloyd-Pottiger deleted the support-inverted-index branch April 10, 2025 13:33

Lloyd-Pottiger restored the support-inverted-index branch April 14, 2025 11:16

Lloyd-Pottiger deleted the support-inverted-index branch April 14, 2025 11:28

breezewish mentioned this pull request Apr 18, 2025

ddl, parser: refine Vector Index and Columnar Index check #60663

Merged

13 tasks

zeminzhou pushed a commit to zeminzhou/tidb that referenced this pull request May 6, 2025

*: support columnar inverted index (pingcap#59571)

853aa6a

close pingcap#59880

budney pushed a commit to budney/tidb that referenced this pull request May 18, 2025

*: support columnar inverted index (pingcap#59571)

d74ce3f

close pingcap#59880

Conversation

Lloyd-Pottiger commented Feb 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

Uh oh!

tiprow bot commented Feb 17, 2025

Uh oh!

wuhuizuo commented Feb 17, 2025

Uh oh!

ti-chi-bot bot commented Feb 17, 2025

Uh oh!

tiprow bot commented Feb 17, 2025

Uh oh!

tiprow bot commented Feb 17, 2025

Uh oh!

codecov bot commented Mar 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tangenta Apr 8, 2025

Choose a reason for hiding this comment

Uh oh!

Lloyd-Pottiger Apr 8, 2025

Choose a reason for hiding this comment

Uh oh!

tangenta left a comment

Choose a reason for hiding this comment

Uh oh!

ti-chi-bot bot commented Apr 8, 2025

[LGTM Timeline notifier]

Uh oh!

winoros Apr 8, 2025

Choose a reason for hiding this comment

Uh oh!

Lloyd-Pottiger Apr 9, 2025

Choose a reason for hiding this comment

Uh oh!

ti-chi-bot bot commented Apr 9, 2025

Uh oh!

Lloyd-Pottiger commented Apr 9, 2025

Uh oh!

tiprow bot commented Apr 9, 2025

Uh oh!

Lloyd-Pottiger commented Apr 10, 2025

Uh oh!

tiprow bot commented Apr 10, 2025

Uh oh!

Lloyd-Pottiger commented Apr 10, 2025

Uh oh!

tiprow bot commented Apr 10, 2025

Uh oh!

Lloyd-Pottiger commented Apr 10, 2025

Uh oh!

tiprow bot commented Apr 10, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Lloyd-Pottiger commented Feb 17, 2025 •

edited

Loading

codecov bot commented Mar 21, 2025 •

edited

Loading