Skip to content

sql: add built-in functions for workload level index recommendations#105017

Merged
craig[bot] merged 1 commit intocockroachdb:masterfrom
qiyanghe1998:workload-level-index-rec-ui
Jul 27, 2023
Merged

sql: add built-in functions for workload level index recommendations#105017
craig[bot] merged 1 commit intocockroachdb:masterfrom
qiyanghe1998:workload-level-index-rec-ui

Conversation

@qiyanghe1998
Copy link
Copy Markdown
Contributor

@qiyanghe1998 qiyanghe1998 commented Jun 15, 2023

sql: add built-in functions for workload level index recommendations

This commit adds 4 generator built-in functions for workload level index
recommendations: workload_index_recs(), workload_index_recs(timestamptz:
TimestampTZ), workload_index_recs(budget: string) and
workload_index_recs(timestamptz: TimestampTZ, budget: string). The return types
of them are all a string column with multiple rows, one row for each index
recommendation. The user can issue a command like select workload_index_recs(...); and then get the index recommendations for the whole
workload.

Only those workload after the given timestamptz will be considered for the
index recommendations. If there is no timestamptz, all the workload stored in
system.statement_statistics will be taken into account. As for the budget, it
represents the space limit for the recommended indexes, e.g. 42GB, 580MiB. If
it is specified, the workload index recommendation engine will prune the size
of all the indexes to satisfy the space constraint.

A method called workloadindexrec.FindWorkloadRecs() will be implemented to
return the array of indexRecs and error message for these 4 generator built-in
functions to invoke.

Release note: None

Epic: None

@qiyanghe1998 qiyanghe1998 requested review from a team as code owners June 15, 2023 22:11
@qiyanghe1998 qiyanghe1998 requested a review from DrewKimball June 15, 2023 22:12
@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

@qiyanghe1998 qiyanghe1998 force-pushed the workload-level-index-rec-ui branch from 0ae3945 to 829f922 Compare June 20, 2023 14:50
Copy link
Copy Markdown
Contributor

@mgartner mgartner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be better to return a string column with multiple rows, one row for each index recommendation, than an array of strings? What do you think? You can look at set-returning built-ins for inspiration: https://github.com/cockroachdb/cockroach/blob/42634711a5869bad8cde5677176cdec7a8f507d0/pkg/sql/sem/builtins/generator_builtins.go

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @DrewKimball)

@qiyanghe1998 qiyanghe1998 force-pushed the workload-level-index-rec-ui branch 2 times, most recently from 1adea4b to 626c990 Compare June 23, 2023 21:22
@qiyanghe1998
Copy link
Copy Markdown
Contributor Author

Yeah, it will be more clear using multiple rows.

@qiyanghe1998 qiyanghe1998 requested a review from mgartner June 23, 2023 21:26
Copy link
Copy Markdown
Contributor

@mgartner mgartner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 2 of 5 files at r2.
Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @DrewKimball and @qiyanghe1998)


pkg/sql/sem/builtins/generator_builtins_test.go line 86 at r2 (raw file):

		require.Equalf(t, false, rows.Next(), "failed test case %d", i+1)
	}
}

It'd be more consistent with the rest of the tests if these were logic tests. Is there a reason not to put them in a logic test like pkg/sql/logictest/testdata/logic_test/workload_recs?

@qiyanghe1998 qiyanghe1998 force-pushed the workload-level-index-rec-ui branch from 626c990 to 6bf494c Compare June 26, 2023 21:16
@qiyanghe1998
Copy link
Copy Markdown
Contributor Author

pkg/sql/sem/builtins/generator_builtins_test.go line 86 at r2 (raw file):

Previously, mgartner (Marcus Gartner) wrote…

It'd be more consistent with the rest of the tests if these were logic tests. Is there a reason not to put them in a logic test like pkg/sql/logictest/testdata/logic_test/workload_recs?

You are right. Fixed.

@qiyanghe1998 qiyanghe1998 requested a review from mgartner June 26, 2023 21:18
@rafiss rafiss removed the request for review from a team July 18, 2023 21:01
Copy link
Copy Markdown
Collaborator

@rytaft rytaft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm: sorry you've been waiting so long for a review! Just a few nits below.

Reviewed 1 of 5 files at r2, 11 of 11 files at r3, all commit messages.
Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @DrewKimball, @mgartner, and @qiyanghe1998)


-- commits line 24 at r3:
I think this should have a release note, since these functions will be user-facing


pkg/sql/sem/builtins/generator_builtins.go line 285 at r3 (raw file):

			tree.ParamTypes{},
			types.String,
			makeWorkloadIndexRecsGeneratorFactory(false, false),

nit: for each of these boolean parameters, we usually add a comment to say what it is. Something like this (here and below):

makeWorkloadIndexRecsGeneratorFactory(false /* hasTs */, false /* hasBgt */),

pkg/sql/sem/builtins/generator_builtins.go line 1121 at r3 (raw file):

// makeWorkloadIndexRecsGeneratorFactory uses the arrayValueGenerator to
// return all the index recommendations as an array of strings
func makeWorkloadIndexRecsGeneratorFactory(hasTs bool, hasBgt bool) eval.GeneratorOverload {

nit: I'd spell these params out: hasTimestamp and hasBudget. Also update the function comment to mention what these params mean.


pkg/sql/sem/builtins/generator_builtins.go line 1123 at r3 (raw file):

func makeWorkloadIndexRecsGeneratorFactory(hasTs bool, hasBgt bool) eval.GeneratorOverload {
	return func(_ context.Context, _ *eval.Context, _ tree.Datums) (eval.ValueGenerator, error) {
		// Invoke the workloadindexrec.FindWorkloadRecs() to get indexRecs, err once it is implemented.

nit: comment should be <= 80 characters wide. you can just wrap the text here.


pkg/sql/logictest/testdata/logic_test/workload_indexrecs line 1 at r3 (raw file):

# get workload index-recs

It would be good to add a comment to the top of this file mentioning that the data returned right now is just dummy data.

@mgartner
Copy link
Copy Markdown
Contributor

-- commits line 24 at r3:

Previously, rytaft (Rebecca Taft) wrote…

I think this should have a release note, since these functions will be user-facing

Should we wait on the release note until they are implemented? They're just stubs in this PR.

@qiyanghe1998 qiyanghe1998 force-pushed the workload-level-index-rec-ui branch from 6bf494c to 34f2628 Compare July 21, 2023 18:08
Copy link
Copy Markdown
Collaborator

@rytaft rytaft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 2 of 2 files at r4, all commit messages.
Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @DrewKimball and @mgartner)


-- commits line 24 at r3:

Previously, mgartner (Marcus Gartner) wrote…

Should we wait on the release note until they are implemented? They're just stubs in this PR.

Fine with me, as long as we don't forget to add one eventually (I think the docs team usually does some massaging anyway, so it's also fine to add something in both places)

@qiyanghe1998
Copy link
Copy Markdown
Contributor Author

bors r+

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Jul 24, 2023

Merge conflict.

@qiyanghe1998 qiyanghe1998 force-pushed the workload-level-index-rec-ui branch from 34f2628 to 6e2986d Compare July 26, 2023 18:59
@qiyanghe1998
Copy link
Copy Markdown
Contributor Author

bors r+

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Jul 26, 2023

Build failed (retrying...):

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Jul 26, 2023

Build failed:

@yuzefovich
Copy link
Copy Markdown
Member

I think this PR is missing ./dev gen docs.

bors r-

This commit adds 4 generator built-in functions for workload level index
recommendations: workload_index_recs(), workload_index_recs(timestamptz:
TimestampTZ), workload_index_recs(budget: string) and
workload_index_recs(timestamptz: TimestampTZ, budget: string). The return types
of them are all a string column with multiple rows, one row for each index
recommendation.  The user can issue a command like `select
workload_index_recs(...);` and then get the index recommendations for the whole
workload.

Only those workload after the given timestamptz will be considered for the
index recommendations. If there is no timestamptz, all the workload stored in
`system.statement_statistics` will be taken into account. As for the budget, it
represents the space limit for the recommended indexes, e.g. 42GB, 580MiB.  If
it is specified, the workload index recommendation engine will prune the size
of all the indexes to satisfy the space constraint.

A method called workloadindexrec.FindWorkloadRecs() will be implemented to
return the array of indexRecs and error message for these 4 generator built-in
functions to invoke.

Release note: None
@qiyanghe1998 qiyanghe1998 force-pushed the workload-level-index-rec-ui branch from 6e2986d to 192d241 Compare July 27, 2023 14:25
@qiyanghe1998
Copy link
Copy Markdown
Contributor Author

bors r+

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Jul 27, 2023

This PR was included in a batch that timed out, it will be automatically retried

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Jul 27, 2023

Build failed (retrying...):

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Jul 27, 2023

Build succeeded:

@craig craig bot merged commit 9d6232e into cockroachdb:master Jul 27, 2023
@qiyanghe1998 qiyanghe1998 deleted the workload-level-index-rec-ui branch July 27, 2023 21:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants