Skip to content

kv: bump timestamp cache to Pushee.MinTimestamp on PUSH_ABORT#60835

Merged
craig[bot] merged 1 commit intocockroachdb:masterfrom
nvb:nvanbenschoten/pushTime
Apr 1, 2021
Merged

kv: bump timestamp cache to Pushee.MinTimestamp on PUSH_ABORT#60835
craig[bot] merged 1 commit intocockroachdb:masterfrom
nvb:nvanbenschoten/pushTime

Conversation

@nvb
Copy link
Copy Markdown
Contributor

@nvb nvb commented Feb 20, 2021

Fixes #60779.
Fixes #60580.

We were only checking that the batch header timestamp was equal to or
greater than this pushee's min timestamp, so this is as far as we can
bump the timestamp cache.

@nvb nvb requested a review from andreimatei February 20, 2021 18:43
@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

Copy link
Copy Markdown
Contributor

@andreimatei andreimatei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We were only checking that the batch header timestamp was equal to or
greater than this pushee's min timestamp...

So? Issues with introducing timestamp in the tscache potentially above the local clock?

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @nvanbenschoten)


pkg/kv/kvserver/replica_tscache.go, line 185 at r1 (raw file):

			} else {
				key = transactionPushMarker(start, pushee.ID)
				pushTS = pushee.WriteTimestamp

so why is this one safe if the other one isn't?

@nvb nvb force-pushed the nvanbenschoten/pushTime branch 2 times, most recently from 65e8855 to fbc6796 Compare March 23, 2021 19:46
Copy link
Copy Markdown
Contributor Author

@nvb nvb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Issues with introducing timestamp in the tscache potentially above the local clock?

Right. Even though we've removed the assertion in #61130, I think we still want the invariant that the timestamp cache doesn't contain non-synthetic timestamps above the local clock, if only for the mixed-version cluster case.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @andreimatei)


pkg/kv/kvserver/replica_tscache.go, line 185 at r1 (raw file):

Previously, andreimatei (Andrei Matei) wrote…

so why is this one safe if the other one isn't?

Because we checked the PushTo timestamp against the local clock in cmd_push_txn.go, but this may be below the pushee's WriteTimestamp for PUSH_ABORT requests. I added a comment.

@nvb
Copy link
Copy Markdown
Contributor Author

nvb commented Mar 23, 2021

@ajstorm found that this fixes one of the issues in #60773.

Fixes cockroachdb#60779.

We were only checking that the batch header timestamp was equal to or
greater than this pushee's min timestamp, so this is as far as we can
bump the timestamp cache.
@nvb
Copy link
Copy Markdown
Contributor Author

nvb commented Mar 29, 2021

Friendly ping.

@ajstorm
Copy link
Copy Markdown
Collaborator

ajstorm commented Mar 30, 2021

Added the GA blocker label, as a GA blocker (#60773) depends on this.

Copy link
Copy Markdown
Contributor

@andreimatei andreimatei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:LGTM:

No easy test to be written?

Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @andreimatei)

@nvb
Copy link
Copy Markdown
Contributor Author

nvb commented Mar 31, 2021

TFTR! This jumps right out on the kvnemesis test I'm about to add, so I'm satisfied with that.

bors r+

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Mar 31, 2021

Build failed:

@nvb
Copy link
Copy Markdown
Contributor Author

nvb commented Mar 31, 2021

Flake in TestLogic/fakedist-spec-planning/deep_interleaving.

bors r+

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Apr 1, 2021

Build succeeded:

@craig craig bot merged commit 41f921d into cockroachdb:master Apr 1, 2021
ajstorm added a commit to ajstorm/cockroach that referenced this pull request Apr 1, 2021
With cockroachdb#60835 merged, this test no longer flakes. I've stressed it on my
GCE worker now for a while an it's all good.

Resolves cockroachdb#60773.

Release note: None
craig bot pushed a commit that referenced this pull request Apr 1, 2021
62954: sql: Re-enable multi_region_backup test r=arulajmani a=ajstorm

With #60835 merged, this test no longer flakes. I've stressed it on my
GCE worker now for a while an it's all good.

Resolves #60773.

Release note: None

62959: sql: lease acquisition of OFFLINE descs may starve bulk operations r=ajwerner a=fqazi

Fixes: #61798

Previously, offline descriptors would never have their leases
cached and they would be released once the reference count hit zero.
This was inadequate because when attempting to online these tables
again the lease acquisition could be pushed back by other operations,
leading to starvation / live locks. To address this, this patch will
allow the leases of offline descriptors to be cached.

Release note (bug fix): Lease acquisitions of descriptor in a
offline state may starve out bulk operations (backup / restore)

Co-authored-by: Adam Storm <storm@cockroachlabs.com>
Co-authored-by: Faizan Qazi <faizan@cockroachlabs.com>
@nvb nvb deleted the nvanbenschoten/pushTime branch April 1, 2021 19:38
ajstorm added a commit to ajstorm/cockroach that referenced this pull request Apr 5, 2021
With cockroachdb#60835 merged, this test no longer flakes. I've stressed it on my
GCE worker now for a while an it's all good.

Resolves cockroachdb#60773.

Release note: None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

4 participants