Skip to content

sql: support nearest_only argument for with_min_timestamp/max_staleness#67837

Merged
craig[bot] merged 1 commit intocockroachdb:masterfrom
otan-cockroach:local_only
Jul 22, 2021
Merged

sql: support nearest_only argument for with_min_timestamp/max_staleness#67837
craig[bot] merged 1 commit intocockroachdb:masterfrom
otan-cockroach:local_only

Conversation

@otan
Copy link
Copy Markdown
Contributor

@otan otan commented Jul 21, 2021

Release note (sql change): Introduced a nearest_only argument for
with_min_timestamp/with_max_staleness, which enforces bounded
staleness reads only talk to the nearest replica.

@otan otan requested a review from nvb July 21, 2021 01:17
@otan otan requested a review from a team as a code owner July 21, 2021 01:17
@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

Copy link
Copy Markdown
Contributor

@nvb nvb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

I'm a little concerned with the ambiguity of the term "local", both because it seems imprecise ("local to what?") and because it won't match the semantics we're planning on giving the flag. I imagine that "local" is meant to mean "with the same region locality tier as the gateway". So that would imply that if no replicas for a range are in the same region as the gateway, then we should always fail. Similarly, it would imply that if two replicas for a range are in the same region as the gateway, then we should try both but not try any other replicas.

That's not quite what we are planning to implement. Instead, what we'll implement will be more aligned with "nearest_only" or "closest_only". We'll use the closest available replica (like the info text mentions) regardless of whether it is in the same region as the gateway, and that's the only replica we'll try to use.

Reviewed 5 of 5 files at r1.
Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @otan)


docs/RFCS/20210519_bounded_staleness_reads.md, line 94 at r1 (raw file):

`local_only`argument  that

nit: the spacing looks off here. Same thing with down below.


docs/RFCS/20210519_bounded_staleness_reads.md, line 175 at r1 (raw file):

if the table descriptor is not available locally

What do you mean by this? Do you mean that we will reject the query if the system.descriptor table's leaseholder is not local and we don't have a lease on the table descriptor at the time of the query?


docs/RFCS/20210519_bounded_staleness_reads.md, line 540 at r1 (raw file):

This is still the case if `local_only` is specified.

Why does local_only have an impact on this? We'll still iterate up the schema history when local_only is set and a newer version is unavailable. It's only when we get to the earliest usable schema version that local_only makes a difference.


pkg/sql/sem/builtins/builtins.go, line 2487 at r1 (raw file):

				return withMinTimestamp(ctx, tree.MustBeDTimestampTZ(args[0]).Time)
			},
			PreferredOverload: true,

Why do both WithMinTimestampFunctionName functions set PreferredOverload to true but both WithMaxStalenessFunctionName leave it as false?

Copy link
Copy Markdown
Contributor Author

@otan otan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i've changed it to nearest_only. i misunderstood it to mean "local" as in local to the region, but the difference you describe makes sense.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @nvanbenschoten)


docs/RFCS/20210519_bounded_staleness_reads.md, line 175 at r1 (raw file):

Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
if the table descriptor is not available locally

What do you mean by this? Do you mean that we will reject the query if the system.descriptor table's leaseholder is not local and we don't have a lease on the table descriptor at the time of the query?

i've clarified this a little bit.


docs/RFCS/20210519_bounded_staleness_reads.md, line 540 at r1 (raw file):

Previously, nvanbenschoten (Nathan VanBenschoten) wrote…
This is still the case if `local_only` is specified.

Why does local_only have an impact on this? We'll still iterate up the schema history when local_only is set and a newer version is unavailable. It's only when we get to the earliest usable schema version that local_only makes a difference.

fixed this up a bit


pkg/sql/sem/builtins/builtins.go, line 2487 at r1 (raw file):

Previously, nvanbenschoten (Nathan VanBenschoten) wrote…

Why do both WithMinTimestampFunctionName functions set PreferredOverload to true but both WithMaxStalenessFunctionName leave it as false?

oops, that doesn't need to be set.
i've removed PreferredOverload mentions.

@otan otan changed the title sql: support local_only argument for with_min_timestamp/max_staleness sql: support nearest_only argument for with_min_timestamp/max_staleness Jul 21, 2021
@otan otan requested a review from nvb July 21, 2021 03:37
Copy link
Copy Markdown
Contributor

@nvb nvb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 4 of 5 files at r2, 1 of 1 files at r3.
Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @nvanbenschoten and @otan)


docs/RFCS/20210519_bounded_staleness_reads.md, line 6 at r1 (raw file):

- Authors: Nathan VanBenschoten, Rebecca Taft, Oliver Tan
- RFC PR: #66020.
- Cockroach Issue: #25405.

nit: while here, want to delete the period above as well?


docs/RFCS/20210519_bounded_staleness_reads.md, line 175 at r1 (raw file):

Previously, otan (Oliver Tan) wrote…

i've clarified this a little bit.

I see. Is this actually what we are planning on implementing though? I wasn't thinking that nearest_only would have any interaction with table leases. It seems like a pretty rough failure mode that if a table descriptor is not leased at some point in time and we only have bounded staleness reads on the table, we'll never grab a lease and begin rejecting all queries.

We assume that the system.descriptor table is always going to be available, so a missing lease will lead to a blip in latency, but I think that's ok since it won't block indefinitely during a region outage. If users truly want a hard latency cap for all operations, they should still set timeouts.

Put another way, I don't think we need to make the nearest_only flag result in all cross-region operations being rejected. I think it's enough to keep it very constrained to its interaction with dynamic timestamp selection.


docs/RFCS/20210519_bounded_staleness_reads.md, line 12 at r3 (raw file):

Bounded staleness reads are a form of historical read-only queries that use a
dynamic, system-determined timestamp, subject to a user-provided staleness
bound, to read nearby local replicas while minimizing data staleness. They

Should we revert this change?


docs/RFCS/20210519_bounded_staleness_reads.md, line 39 at r3 (raw file):

but avoids the use of the leaseholder.

It doesn't even necessarily do this. The leaseholder can be used if it is the closest replica.


pkg/sql/sem/builtins/builtins.go, line 7649 at r3 (raw file):

const nearestOnlyInfo = `

If nearest_only is set to true, reads that cannot be served using a nearby replica

s/a nearby replica/the nearest available replica/

…ness

Release note (sql change): Introduced a `nearest_only` argument for
`with_min_timestamp`/`with_max_staleness`, which enforces bounded
staleness reads only talk to the nearest replica.
Copy link
Copy Markdown
Contributor Author

@otan otan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @nvanbenschoten)


docs/RFCS/20210519_bounded_staleness_reads.md, line 175 at r1 (raw file):

Previously, nvanbenschoten (Nathan VanBenschoten) wrote…

I see. Is this actually what we are planning on implementing though? I wasn't thinking that nearest_only would have any interaction with table leases. It seems like a pretty rough failure mode that if a table descriptor is not leased at some point in time and we only have bounded staleness reads on the table, we'll never grab a lease and begin rejecting all queries.

We assume that the system.descriptor table is always going to be available, so a missing lease will lead to a blip in latency, but I think that's ok since it won't block indefinitely during a region outage. If users truly want a hard latency cap for all operations, they should still set timeouts.

Put another way, I don't think we need to make the nearest_only flag result in all cross-region operations being rejected. I think it's enough to keep it very constrained to its interaction with dynamic timestamp selection.

i see. so system.descriptor is always available on some replica is the assumption, i think that's reasonable. i was confusing it with my initial interpretation of local.


docs/RFCS/20210519_bounded_staleness_reads.md, line 12 at r3 (raw file):

Previously, nvanbenschoten (Nathan VanBenschoten) wrote…

Should we revert this change?

oops

Copy link
Copy Markdown
Contributor

@nvb nvb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 3 of 3 files at r4.
Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @otan)

@otan
Copy link
Copy Markdown
Contributor Author

otan commented Jul 21, 2021

bors r=nvanbenschoten

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Jul 21, 2021

Build failed (retrying...):

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Jul 22, 2021

Build succeeded:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants