Skip to content

sql: use CommandResult for displaying ParameterStatus updates#45140

Merged
craig[bot] merged 1 commit intocockroachdb:masterfrom
otan-cockroach:cr_try_2
Mar 3, 2020
Merged

sql: use CommandResult for displaying ParameterStatus updates#45140
craig[bot] merged 1 commit intocockroachdb:masterfrom
otan-cockroach:cr_try_2

Conversation

@otan
Copy link
Copy Markdown
Contributor

@otan otan commented Feb 17, 2020

We were previously sending ParameterStatus using a listener. This
is refactored to talk using CommandResult instead, with results only
being buffered if "Close()" is called. To do this, a new
AppendParamStatusUpdate function to RestrictedCommandResult.

Since setting vars live in sessionDataMutator, we add a new
paramStatusUpdater interface which uses the RestrictedCommandResult.
This var is reset each time we execute with an openTxn with the
CommandResult used for it.

Also renamed "StatusParam" to "ParamStatus", which is closer to the name
"ParameterStatus".

Release note: None

@otan otan requested a review from andreimatei February 17, 2020 08:17
@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

@otan otan force-pushed the cr_try_2 branch 3 times, most recently from d02abdf to 8095486 Compare February 23, 2020 21:53
Copy link
Copy Markdown
Contributor

@andreimatei andreimatei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: No need for this in the commit msg "No release note as there is no functional difference."
Also "The existing pgwire tests still pass." Of course they pass, otherwise you wouldn't be sending the PR :).

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @andreimatei and @otan)


pkg/sql/conn_executor.go, line 1651 at r1 (raw file):

// and writing up to the CommandComplete message.
func (ex *connExecutor) execCopyIn(
	ctx context.Context, cmd CopyIn, res CommandResultCommBase,

do you really need this res?


pkg/sql/conn_executor_exec.go, line 825 at r1 (raw file):

// non-nil historicalTimestamp implies a ReadOnly rwMode.
func (ex *connExecutor) beginTransactionTimestampsAndReadMode(
	ctx context.Context, s *tree.BeginTransaction, res RestrictedCommandResult,

do you really need the res?


pkg/sql/conn_executor_prepare.go, line 175 at r1 (raw file):

	ex.statsCollector.reset(&ex.server.sqlStats, ex.appStats, &ex.phaseTimes)
	p := &ex.planner
	ex.resetPlanner(ctx, p, txn, ex.server.cfg.Clock.PhysicalTime() /* stmtTS */, stmt.NumAnnotations, res)

long line


pkg/sql/conn_executor_prepare.go, line 175 at r1 (raw file):

	ex.statsCollector.reset(&ex.server.sqlStats, ex.appStats, &ex.phaseTimes)
	p := &ex.planner
	ex.resetPlanner(ctx, p, txn, ex.server.cfg.Clock.PhysicalTime() /* stmtTS */, stmt.NumAnnotations, res)

Does this guy ever need this res? Give it a no-op one and remove the new method arg and above.


pkg/sql/conn_io.go, line 635 at r1 (raw file):

}

// CommandResultCommBase represents communication with the client for

This name...
How about ParamStatusSender?


pkg/sql/conn_io.go, line 877 at r1 (raw file):

// BufferParamStatus is part of the RestrictedCommandResult interface.
func (r *bufferedCommandResult) BufferParamStatus(key string, val string) {}

I'd say either accumulate the key/value and give access to it, or panic("unimplemented")


pkg/sql/exec_util.go, line 1793 at r1 (raw file):

	defaults SessionDefaults
	settings *cluster.Settings
	// commResultComm communicates results through the CommandResult to the client.

name in comment doesn't match

Copy link
Copy Markdown
Contributor Author

@otan otan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @andreimatei)


pkg/sql/conn_executor.go, line 1651 at r1 (raw file):

Previously, andreimatei (Andrei Matei) wrote…

do you really need this res?

the base, yes. I want to send notices at one point (master...otan-cockroach:buf_notice_2).


pkg/sql/conn_executor_exec.go, line 825 at r1 (raw file):

Previously, andreimatei (Andrei Matei) wrote…

do you really need the res?

yes, see above.


pkg/sql/conn_executor_prepare.go, line 175 at r1 (raw file):

Previously, andreimatei (Andrei Matei) wrote…

long line

Done.


pkg/sql/conn_executor_prepare.go, line 175 at r1 (raw file):

Previously, andreimatei (Andrei Matei) wrote…

Does this guy ever need this res? Give it a no-op one and remove the new method arg and above.

yes, because I want to be able to send notices through this one too.


pkg/sql/conn_io.go, line 635 at r1 (raw file):

Previously, andreimatei (Andrei Matei) wrote…

This name...
How about ParamStatusSender?

I do want to re-use this for master...otan-cockroach:buf_notice_2, so ParamStatusSender won't do.


pkg/sql/conn_io.go, line 877 at r1 (raw file):

Previously, andreimatei (Andrei Matei) wrote…

I'd say either accumulate the key/value and give access to it, or panic("unimplemented")

Done.


pkg/sql/exec_util.go, line 1793 at r1 (raw file):

Previously, andreimatei (Andrei Matei) wrote…

name in comment doesn't match

Done.

Copy link
Copy Markdown
Contributor

@andreimatei andreimatei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @andreimatei and @otan)


pkg/sql/conn_executor.go, line 1651 at r1 (raw file):

Previously, otan (Oliver Tan) wrote…

the base, yes. I want to send notices at one point (master...otan-cockroach:buf_notice_2).

future plans should generally be left to the future :)
But even with that PR - do you really need to send notices from execCopyIn?

Copy link
Copy Markdown
Contributor Author

@otan otan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @andreimatei)


pkg/sql/conn_executor.go, line 1651 at r1 (raw file):

Previously, andreimatei (Andrei Matei) wrote…

future plans should generally be left to the future :)
But even with that PR - do you really need to send notices from execCopyIn?

I mean I'd rather leave that option open haha - seems annoying to debug if someone is adding it. Flexible either way; if you feel strongly about it I'll make it be silent instead.

Copy link
Copy Markdown
Contributor

@andreimatei andreimatei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @andreimatei and @otan)


pkg/sql/conn_executor.go, line 1651 at r1 (raw file):

Previously, otan (Oliver Tan) wrote…

I mean I'd rather leave that option open haha - seems annoying to debug if someone is adding it. Flexible either way; if you feel strongly about it I'll make it be silent instead.

I see this passing of res everywhere as significant pollution, so I'd much rather keep it to a minimum. Because the more places we pass it through, the more I'd have to litigate the name - res doesn't seem appropriate.
But see below, maybe these arguments can go away from everywhere and then this point might be moot.


pkg/sql/conn_executor.go, line 2083 at r2 (raw file):

	stmtTS time.Time,
	numAnnotations tree.AnnotationIdx,
	res CommandResultCommBase,

I'd take out this arg from resetPlanner() unless there was a particular reason to put it here. You want to modify the sessionDataMutator, not the planner. The fact that the planner even knows about the sessionDataMutator is unfortunate. I'd assign p.sessionDataMutator explicitly at the beginning of executing each statement (in execStmtInOpenState() or such) - and then I think the proliferation of res everywhere can stop.

Copy link
Copy Markdown
Contributor Author

@otan otan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @andreimatei and @otan)


pkg/sql/conn_executor.go, line 1651 at r1 (raw file):

Previously, andreimatei (Andrei Matei) wrote…

I see this passing of res everywhere as significant pollution, so I'd much rather keep it to a minimum. Because the more places we pass it through, the more I'd have to litigate the name - res doesn't seem appropriate.
But see below, maybe these arguments can go away from everywhere and then this point might be moot.

I can change the name, but I need it in the planner to send notices. I guess the split makes it seem unnecessary but it is in the immediate follow up.

Copy link
Copy Markdown
Contributor

@andreimatei andreimatei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @andreimatei and @otan)


pkg/sql/conn_executor.go, line 1651 at r1 (raw file):

Previously, otan (Oliver Tan) wrote…

I can change the name, but I need it in the planner to send notices. I guess the split makes it seem unnecessary but it is in the immediate follow up.

We really need to stop putting execution-related stuff in the planner. Can you put it in runParams instead?

If you want me to have more context than what's in this commit, I think that's a sign you've split the commits wrong. Just squash them.

Copy link
Copy Markdown
Contributor Author

@otan otan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @andreimatei)


pkg/sql/conn_executor.go, line 1651 at r1 (raw file):

Previously, andreimatei (Andrei Matei) wrote…

We really need to stop putting execution-related stuff in the planner. Can you put it in runParams instead?

If you want me to have more context than what's in this commit, I think that's a sign you've split the commits wrong. Just squash them.

Seems like I have to thread that through to DistSQL / Planning context. Is that okay? It also seems much more helpful to have it in a place more universally easy to access (i.e. my original desire to have it in the evalCtx), but it seems we prefer it as locked down as possible (which is fine, the abstraction boundaries and desires of others I'm, less familiar with).

Seemed like this PR's would make sense on its own with some futureproofing for notices I was trying to do but looks like it didn't play out. We can discuss over it when I put the next PR over it.


pkg/sql/conn_executor.go, line 2083 at r2 (raw file):

Previously, andreimatei (Andrei Matei) wrote…

I'd take out this arg from resetPlanner() unless there was a particular reason to put it here. You want to modify the sessionDataMutator, not the planner. The fact that the planner even knows about the sessionDataMutator is unfortunate. I'd assign p.sessionDataMutator explicitly at the beginning of executing each statement (in execStmtInOpenState() or such) - and then I think the proliferation of res everywhere can stop.

I can remove the controversy of putting this inside resetPlanner in this PR, but it seemed to be killing two birds with one stone.

If it's controversial I can take it out for this PR and do the suggestion of setting p.sessionDataMutator explicitly at the beginning of executing each statement (in execStmtInOpenState() or such). Personally feels like it's dirtier and more error prone, since you could mutate the session somewhere else by accident and not have the param status sent properly, but /shrug

@otan otan force-pushed the cr_try_2 branch 2 times, most recently from 918671f to 1a4cf16 Compare February 28, 2020 22:30
@otan
Copy link
Copy Markdown
Contributor Author

otan commented Feb 29, 2020

stripped out all the passing of res into everything, keeping it CommandResultCommBase so i can share the interface with BufferNotice, but if you feel strongly can name it ParamStatusSender instead, and have a separate interface for BufferNotice in the near future.

Copy link
Copy Markdown
Contributor

@andreimatei andreimatei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @andreimatei and @otan)


pkg/sql/conn_io.go, line 635 at r1 (raw file):

Previously, otan (Oliver Tan) wrote…

I do want to re-use this for master...otan-cockroach:buf_notice_2, so ParamStatusSender won't do.

I wouldn't introduce a new interface here for this. I'd define the method in RestrictedCommandResult, and then I'd define the interface in sql as ParamStatusSender. And when the notices come, I'd define another interface for that also in sql - Noticer or something.
The fact that the comment on this interface seems nonsensical I think is a hint that some abstraction is not quite right: a command result already "represents communication with the client".


pkg/sql/conn_io.go, line 640 at r3 (raw file):

	// BufferParamStatus buffers any updates to a parameter status.
	// This gets flushed only when the CommandResult is closed.
	BufferParamStatus(string, string)

I think SetParamStatus would be better. The buffering aspects of the command results are hidden from clients; that's what the rewind capabilities are supposed to abstract.

@otan
Copy link
Copy Markdown
Contributor Author

otan commented Mar 2, 2020


pkg/sql/conn_io.go, line 640 at r3 (raw file):

Previously, andreimatei (Andrei Matei) wrote…

I think SetParamStatus would be better. The buffering aspects of the command results are hidden from clients; that's what the rewind capabilities are supposed to abstract.

hmm, what about AddParamStatusUpdate? the problem for me is Set ignores the fact that multiple param status updates may be sent (even for the same key - which I think is possible).

Copy link
Copy Markdown
Contributor

@andreimatei andreimatei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @andreimatei and @otan)


pkg/sql/conn_io.go, line 640 at r3 (raw file):

Previously, otan (Oliver Tan) wrote…

hmm, what about AddParamStatusUpdate? the problem for me is Set ignores the fact that multiple param status updates may be sent (even for the same key - which I think is possible).

sgtm

We were previously sending ParameterStatus using a listener. This
is refactored to talk using CommandResult instead, with results only
being buffered if "Close()" is called. To do this, a new
`AppendParamStatusUpdate` function to `RestrictedCommandResult`.

Since setting vars live in `sessionDataMutator`, we add a new
`paramStatusUpdater` interface which uses the `RestrictedCommandResult`.
This var is reset each time we execute with an openTxn with the
CommandResult used for it.

Also renamed "StatusParam" to "ParamStatus", which is closer to the name
"ParameterStatus".

Release note: None
Copy link
Copy Markdown
Contributor Author

@otan otan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @andreimatei)


pkg/sql/conn_io.go, line 635 at r1 (raw file):

Previously, andreimatei (Andrei Matei) wrote…

I wouldn't introduce a new interface here for this. I'd define the method in RestrictedCommandResult, and then I'd define the interface in sql as ParamStatusSender. And when the notices come, I'd define another interface for that also in sql - Noticer or something.
The fact that the comment on this interface seems nonsensical I think is a hint that some abstraction is not quite right: a command result already "represents communication with the client".

Done.


pkg/sql/conn_io.go, line 640 at r3 (raw file):

Previously, andreimatei (Andrei Matei) wrote…

sgtm

Done.

@andreimatei
Copy link
Copy Markdown
Contributor

andreimatei commented Mar 2, 2020 via email

@otan
Copy link
Copy Markdown
Contributor Author

otan commented Mar 3, 2020

bors r=andreimatei

@otan
Copy link
Copy Markdown
Contributor Author

otan commented Mar 3, 2020

bors r=andreimatei

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Mar 3, 2020

Build succeeded

@craig craig bot merged commit 06989f3 into cockroachdb:master Mar 3, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants