Skip to content

sql: implement SET LOCAL#69224

Merged
craig[bot] merged 9 commits intocockroachdb:masterfrom
otan-cockroach:local_role
Aug 27, 2021
Merged

sql: implement SET LOCAL#69224
craig[bot] merged 9 commits intocockroachdb:masterfrom
otan-cockroach:local_role

Conversation

@otan
Copy link
Copy Markdown
Contributor

@otan otan commented Aug 23, 2021

Resolves: #32562
Release justification: high priority bug fix

First 3 commits from #69394

@otan otan added the do-not-merge bors won't merge a PR with this label. label Aug 23, 2021
@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

@otan
Copy link
Copy Markdown
Contributor Author

otan commented Aug 23, 2021

Couple of things to assess before this can actually get in:

  • param status updates must be tracked and reverted on commit/rollback.
  • is_superuser needs updates
  • nested transactions
  • runtimeset, setwithplanner
  • the sessiondatastack should probably be some pointer.
  • the mutator factory's base struct may need some rejigging, e.g. DEFAULT evaluation shouldn't need topMutator
  • we should look at how good that "copy" of session data is. i suspect stuff like search_path won't be so nice.
  • tests tests tests

@otan otan added the X-noremind Bots won't notify about PRs with X-noremind label Aug 23, 2021
@otan
Copy link
Copy Markdown
Contributor Author

otan commented Aug 23, 2021

according to #32562 (comment) there is also a txn state on the connexecutor we need to keep a track of. afaict, this is savepoint, but not sure how re-usable that is for this.

Copy link
Copy Markdown
Collaborator

@rafiss rafiss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is looking good! it's actually less messy than i thought it would be..

what's your concern with search_path? is it special in some way?

(btw, we should make sure this also works with SET LOCAL ROLE -- looks like it should, but thats another thing to test)

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @otan)


pkg/sql/conn_executor_exec.go, line 820 at r5 (raw file):

	ex.phaseTimes.SetSessionPhaseTime(sessionphase.SessionEndTransactionCommit, timeutil.Now())
	if !ex.implicitTxn() {
		ex.sessionDataStack = ex.sessionDataStack.Pop()

so i guess for the param status stuff, we'd need a check (maybe inside of Pop that will send out param status messages for all the variables that need it.


pkg/sql/conn_executor_exec.go, line 1248 at r5 (raw file):

) (_ fsm.Event, payload fsm.EventPayload) {
	switch s := ast.(type) {
	case *tree.BeginTransaction:

i think we also need something similar when starting a new SAVEPOINT. (that's the only way to get a nested txn. if you do BEGIN inside an explicit txn, it's an error.)


pkg/sql/exec_util.go, line 2395 at r4 (raw file):

func (noopParamStatusUpdater) BufferParamStatusUpdate(string, string) {}

// sessionDataMutatorFactory creates sessionMutators.

the comment could include why it's needed and also how it interacts with the session data stack


pkg/sql/planner.go, line 347 at r4 (raw file):

	p.extendedEvalCtx = internalExtendedEvalCtx(
		ctx, sd, &sessionDataMutator{

nit: seemed a bit easier to read when it wasn't inlined


pkg/sql/set_var.go, line 145 at r5 (raw file):

	}
	if n.local {
		if params.p.EvalContext().TxnImplicit {

let's match the Postgres warning: "SET LOCAL can only be used in transaction blocks"

weirdly, postgres shows that warning, but then still lets the SET LOCAL go ahead as if it were a normal SET.


pkg/sql/parser/sql.y, line 4432 at r5 (raw file):

| set_exprs_internal   { /* SKIP DOC */ }
| SET CONSTRAINTS error { return unimplemented(sqllex, "set constraints") }
| SET LOCAL set_rest

hm i think this works better with a new rule so that we can have the right help text

// %Help: SET LOCAL - change a session variable scoped to the current transaction
// %Category: Cfg
// %Text:
// SET LOCAL <var> { TO | = } <values...>
// SET LOCAL TIME ZONE <tz>
//
// %SeeAlso: SHOW SESSION, RESET, DISCARD, SHOW, SET CLUSTER SETTING, SET TRANSACTION, SET SESSION,
// WEBDOCS/set-vars.html
set_local_stmt:
  SET LOCAL set_rest
  {
    // ...
  }

also, what happens if we put this under preparable_set_stmt?


pkg/sql/sem/tree/eval.go, line 3458 at r3 (raw file):

type EvalContext struct {
	// Session variables. This is a read-only copy of the values owned by the
	// Session. A new element is added to the stack for each explicit transaction.

what is "for each explicit transaction" referring to? i believe it's when a SAVEPOINT is created right?


pkg/sql/sessiondata/session_data.go, line 183 at r1 (raw file):

}

// SessionDataStack represents a stack of SessionData objects.

it would be helpful for the comment to say why we need it -- to support SET LOCAL and txn-scoped session vars


pkg/sql/sessiondata/session_data.go, line 195 at r2 (raw file):

// NewSessionDataStack creates a new SessionDataStack.
func NewSessionDataStack(sds ...*git difSessionData) SessionDataStack {

typo on git diff

@otan
Copy link
Copy Markdown
Contributor Author

otan commented Aug 24, 2021

what's your concern with search_path? is it special in some way?

we need a deep copy of it, basically. otherwise it re-uses the same slice

i'll work on fixing this up if we decide to proceed with SET LOCAL.

@rafiss
Copy link
Copy Markdown
Collaborator

rafiss commented Aug 24, 2021

yeah let's go ahead and try to finish this. @ajstorm and @vy-ton also feel like this would be a good item to complete for 21.2

@otan otan force-pushed the local_role branch 2 times, most recently from 53ee403 to db3d7c9 Compare August 25, 2021 03:04
Copy link
Copy Markdown
Contributor

@ajwerner ajwerner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Drive-by commentary because I was curious.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @otan)


pkg/sql/exec_util.go, line 2488 at r10 (raw file):

}

// sessionDataMutator is the interface used by sessionVars to change the session

nit: this isn't an interface


pkg/sql/sessiondata/session_data.go, line 196 at r10 (raw file):


	// Use an internal variable to prevent abstraction leakage.
	stack []*SessionData

Why not a linked list? How do you feel about:

type Stack struct {
    cur   SessionData // could see making this a pointer
    next *Stack
}

Copy link
Copy Markdown
Contributor Author

@otan otan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @ajwerner, @otan, and @rafiss)


pkg/sql/exec_util.go, line 2395 at r4 (raw file):

Previously, rafiss (Rafi Shamim) wrote…

the comment could include why it's needed and also how it interacts with the session data stack

Done.


pkg/sql/sessiondata/session_data.go, line 195 at r2 (raw file):

Previously, rafiss (Rafi Shamim) wrote…

typo on git diff

Done.


pkg/sql/sessiondata/session_data.go, line 196 at r10 (raw file):

Previously, ajwerner wrote…

	// Use an internal variable to prevent abstraction leakage.
	stack []*SessionData

Why not a linked list? How do you feel about:

type Stack struct {
    cur   SessionData // could see making this a pointer
    next *Stack
}

was thinking it'd re-use the same slice space and because there are certain PopN/fetching all items in the stack as a slice operations i'd like slicing for.

@otan otan force-pushed the local_role branch 6 times, most recently from 51c0ae4 to 34a0d56 Compare August 26, 2021 03:11
@otan otan changed the title [WIP] sql: implement SET LOCAL sql: implement SET LOCAL Aug 26, 2021
@otan otan removed do-not-merge bors won't merge a PR with this label. X-noremind Bots won't notify about PRs with X-noremind labels Aug 26, 2021
@otan otan marked this pull request as ready for review August 26, 2021 03:20
@otan otan requested review from a team and pbardea and removed request for a team August 26, 2021 03:20
@otan otan force-pushed the local_role branch 2 times, most recently from f542d46 to 32bc99f Compare August 26, 2021 04:00
Copy link
Copy Markdown
Collaborator

@rafiss rafiss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

release note nit: the release note should describe how it works with savepoints (SET LOCAL isn't rolled back after releasing a savepoint)

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @ajwerner, @otan, @pbardea, and @rafiss)


pkg/sql/conn_executor_savepoints.go, line 140 at r20 (raw file):

	// We will restore the currSessionData on it, as the SET LOCAL parameters
	// are still kept.
	// We do not have to report param status updates as the SessionData

wait why don't we report them? based on the tests below it looked like the timezone was changing even after doing a RELEASE


pkg/sql/set_var.go, line 32 at r17 (raw file):

)

// setVarNode represents a SET SESSION statement.

nit: represents a SET { LOCAL | SESSION } statement


pkg/sql/set_var.go, line 139 at r19 (raw file):

	}

	if n.v.RuntimeSet != nil {

could you add a comment saying why RuntimeSet and SetWithPlanner don't work with applyOnSessionDataMutators?


pkg/sql/logictest/testdata/logic_test/set_local, line 38 at r17 (raw file):

2020-08-25 08:16:17.123456 -0700 PDT  P1DT15H16M17.123456S

# Rollback and ensure the values are correct.

i think this comment also deserves a link to #69396 mentioning that CRDB diverges from postgres


pkg/sql/parser/testdata/set, line 520 at r16 (raw file):


parse
SET LOCAL TIME ZONE +3

can we test SET LOCAL ROLE bob as well?


pkg/sql/sessiondata/session_data.go, line 56 at r17 (raw file):

func (s *SessionData) Clone() *SessionData {
	// Currently clone does a shallow copy of everything - we can get away with it
	// as all the slices/maps does a copy if it mutates OR are idempotent

nit: i don't think idempotent is the right word. maybe it could just say "operations that affect the whole SessionDataStack"


pkg/sql/sessiondata/session_data.go, line 58 at r17 (raw file):

	// as all the slices/maps does a copy if it mutates OR are idempotent
	// operations (e.g. setting SequenceState should be the setting the
	// same value across all copied sessions).

nit: "across all copied SessionData"

Copy link
Copy Markdown
Contributor Author

@otan otan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @ajwerner, @pbardea, and @rafiss)


pkg/sql/conn_executor_savepoints.go, line 140 at r20 (raw file):

Previously, rafiss (Rafi Shamim) wrote…

wait why don't we report them? based on the tests below it looked like the timezone was changing even after doing a RELEASE

whereabouts? it's because RELEASE doesn't actually commit or rollback a transaction; they just scrub previous savepoints.


pkg/sql/set_var.go, line 139 at r19 (raw file):

Previously, rafiss (Rafi Shamim) wrote…

could you add a comment saying why RuntimeSet and SetWithPlanner don't work with applyOnSessionDataMutators?

Done.


pkg/sql/logictest/testdata/logic_test/set_local, line 38 at r17 (raw file):

Previously, rafiss (Rafi Shamim) wrote…

i think this comment also deserves a link to #69396 mentioning that CRDB diverges from postgres

Done.


pkg/sql/parser/sql.y, line 4432 at r5 (raw file):

Previously, rafiss (Rafi Shamim) wrote…

hm i think this works better with a new rule so that we can have the right help text

// %Help: SET LOCAL - change a session variable scoped to the current transaction
// %Category: Cfg
// %Text:
// SET LOCAL <var> { TO | = } <values...>
// SET LOCAL TIME ZONE <tz>
//
// %SeeAlso: SHOW SESSION, RESET, DISCARD, SHOW, SET CLUSTER SETTING, SET TRANSACTION, SET SESSION,
// WEBDOCS/set-vars.html
set_local_stmt:
  SET LOCAL set_rest
  {
    // ...
  }

also, what happens if we put this under preparable_set_stmt?

Done.


pkg/sql/parser/testdata/set, line 520 at r16 (raw file):

Previously, rafiss (Rafi Shamim) wrote…

can we test SET LOCAL ROLE bob as well?

Done.


pkg/sql/sessiondata/session_data.go, line 183 at r1 (raw file):

Previously, rafiss (Rafi Shamim) wrote…

it would be helpful for the comment to say why we need it -- to support SET LOCAL and txn-scoped session vars

Done.


pkg/sql/sem/tree/eval.go, line 3458 at r3 (raw file):

Previously, rafiss (Rafi Shamim) wrote…

what is "for each explicit transaction" referring to? i believe it's when a SAVEPOINT is created right?

Done.

Release justification: fix for new feature

Release note: None
@otan otan force-pushed the local_role branch 2 times, most recently from ad1ce5a to 0c87ad9 Compare August 27, 2021 05:31
@otan
Copy link
Copy Markdown
Contributor Author

otan commented Aug 27, 2021

bors r=rafiss

craig bot pushed a commit that referenced this pull request Aug 27, 2021
69224: sql: implement SET LOCAL r=rafiss a=otan

Resolves: #32562
Release justification: high priority bug fix

First 3 commits from #69394

Co-authored-by: Oliver Tan <otan@cockroachlabs.com>
The BEGIN and PARSE race here, triggered by the LOCAL sessiondata
tests.

The race is actually as follows:
* BEGIN succeeds and processes in `processCommandsAsync`
* PARSE attempts to access UnqualifiedIntSize from the SessionData,
  before putting the item in the statement buffer. However, with the
  SESSION LOCAL implementation, we Push into the sessiondata.Stack,
  causing a race to trigger between pushing into the stack and reading
  the UnqualifiedIntSize.

This definitely can exist as a race today - some user sets the int_size
to a different number AND sends a PARSE assuming the int size has
changed, but if processCommandsAsync does not process this, it can fail.

This commit temporarily papers over this race for
TestTrimSuspendedPortals by waiting for the BEGIN to
execute before sending the PARSE which reads the int size.

We also skip the tests in PGTest as they can similar hit this race (for
now).

Release justification: test only change

Release note: None
@craig
Copy link
Copy Markdown
Contributor

craig bot commented Aug 27, 2021

Canceled.

@otan
Copy link
Copy Markdown
Contributor Author

otan commented Aug 27, 2021

bors r=rafiss

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Aug 27, 2021

Build failed (retrying...):

otan added 6 commits August 27, 2021 18:38
Release justification: low risk high pri change

Release note: None
This implements SET LOCAL on simple transactions (i.e. BEGIN ...
COMMIT/ROLLBACK, no SAVEPOINTs).

Release justification: high pri feature

Release note (sql change): Introduce SET LOCAL, which sets a session
variable for the duration of the transaction. SET LOCAL is a no-op
outside the transaction.
This commit introduces SET LOCAL for savepoints. The tricky bit here is
to restore an additional element from the stack after rolling back, as
well as being to rollback multiple savepoints at a time.

Release justification: high pri feature, low risk

Release note (sql change): SET LOCAL now works for SAVEPOINTs. ROLLBACK
will rollback any variables set during SET LOCAL. RELEASE TO SAVEPOINT
will continue to use the variables set by SET LOCAL in the transaction.
This commit refactors RuntimeSet and SetWithPlanner to take in a local
argument, so callers must take that into account. Also remove
SetWithPlanner for database - I don't think it's needed.

Release justification: high pri fix for new functionality

Release note: None
This commit adds ParamStatusUpdate changes when a transaction completes
or has a savepoint modified.

We need to add a `GetWithSessionData` method, as extendedEvalContext is
not available at the connExecutor level. We also need to init
`SetWithPlanner` later, because it would introduce an interesting
"initialization loop" error.

Release justification: high priority feature

Release note: None
Release justification: naming change only

Release note: None
@craig
Copy link
Copy Markdown
Contributor

craig bot commented Aug 27, 2021

Canceled.

@otan
Copy link
Copy Markdown
Contributor Author

otan commented Aug 27, 2021

rebase hell

bors r=rafiss

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Aug 27, 2021

Build succeeded:

@craig craig bot merged commit 4bc79bc into cockroachdb:master Aug 27, 2021
craig bot pushed a commit that referenced this pull request Aug 29, 2021
69355: sql: properly populate is_superuser r=rafiss a=otan

Release justification: low risk high benefit change

Last commit only. First two commits in #69224

See individual commits for details.

Co-authored-by: Oliver Tan <otan@cockroachlabs.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

sql: support SET LOCAL and txn-scoped session variable changes

4 participants