Skip to content

storage: Update timestamp cache before updating a lease on a replica#17109

Merged
m-schneider merged 1 commit intocockroachdb:masterfrom
m-schneider:leasechangefix
Jul 20, 2017
Merged

storage: Update timestamp cache before updating a lease on a replica#17109
m-schneider merged 1 commit intocockroachdb:masterfrom
m-schneider:leasechangefix

Conversation

@m-schneider
Copy link
Copy Markdown
Contributor

  • Before we would update the in-memory lease on a replica,
    unlock it and then we would update the timestamp cache. This would
    allow a replica to start processing requests for a lease without
    first having taken into account reads served from the old lease
    holder. We could see this by running:
    make stress PKG=./pkg/sql/logictest
    TESTS=TestParallel/subquery_retry_multinode STRESSFLAGS='-p 32'

This would fail because we would have duplicate values in a table
where inserts should have had monotonically increasing values.

  • After this change the test passes. We also updated splitPostApply,
    to send the current lease as the previous lease to leasePostApply to
    prevent an update to the timestamp cache when a lease hasn't changed
    hands.

Closes #16610

@m-schneider m-schneider requested a review from tbg July 19, 2017 18:27
@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

@tbg
Copy link
Copy Markdown
Member

tbg commented Jul 19, 2017

Looks good so far!


Reviewed 2 of 2 files at r1.
Review status: all files reviewed at latest revision, 2 unresolved discussions, all commit checks successful.


pkg/storage/store.go, line 1861 at r1 (raw file):

	// the PushTxnQueue. Note that we pass in an right lease for prevLease so
	// that we don't unnecessarily update the timestamp cache.
	rightRng.leasePostApply(ctx, rightLease, rightReplicaID, rightLease)

Can you just drop the last argument to leasePostApply and instead load prevLease in leasePostApply? In this callsite, we already have rightRng.Lease == rightLease so it will continue to be a noop, and similarly the other call site won't have to fetch the lease manually. I think the same holds for the ReplicaID which in both cases is picked from mu which you may as well do inside the function.


pkg/storage/store.go, line 1861 at r1 (raw file):

	// the PushTxnQueue. Note that we pass in an right lease for prevLease so
	// that we don't unnecessarily update the timestamp cache.
	rightRng.leasePostApply(ctx, rightLease, rightReplicaID, rightLease)

Looking through the git blame for this call to leasePostApply, I found this: #14305

It seems that the original call intentionally wanted things to look like the lease was changing hands so that pushTxnQueue.Clear() is called in leasePostApply. ISTM that we've changed this back now; is the test that was added in that PR not failing? Can you check what's going on there?


Comments from Reviewable

@m-schneider
Copy link
Copy Markdown
Contributor Author

Review status: 0 of 2 files reviewed at latest revision, 2 unresolved discussions.


pkg/storage/store.go, line 1861 at r1 (raw file):

Previously, tschottdorf (Tobias Schottdorf) wrote…

Looking through the git blame for this call to leasePostApply, I found this: #14305

It seems that the original call intentionally wanted things to look like the lease was changing hands so that pushTxnQueue.Clear() is called in leasePostApply. ISTM that we've changed this back now; is the test that was added in that PR not failing? Can you check what's going on there?

I reran the storage tests and nothing fails. I read through all of the comments for that PR as well and the main concern was for the pushTxnQueue to be enabled after the call to leasePostApply, which it is because that's only gated by iAmTheLeaseHolder. The tests added in that PR only check to see if the queue is enabled, but don't test the contents.


pkg/storage/store.go, line 1861 at r1 (raw file):

Previously, tschottdorf (Tobias Schottdorf) wrote…

Can you just drop the last argument to leasePostApply and instead load prevLease in leasePostApply? In this callsite, we already have rightRng.Lease == rightLease so it will continue to be a noop, and similarly the other call site won't have to fetch the lease manually. I think the same holds for the ReplicaID which in both cases is picked from mu which you may as well do inside the function.

Good point, cleaned up.


Comments from Reviewable

@tbg
Copy link
Copy Markdown
Member

tbg commented Jul 20, 2017

:lgtm:


Reviewed 2 of 2 files at r2.
Review status: all files reviewed at latest revision, 1 unresolved discussion, some commit checks failed.


pkg/storage/store.go, line 1861 at r1 (raw file):

Previously, m-schneider (Masha Schneider) wrote…

I reran the storage tests and nothing fails. I read through all of the comments for that PR as well and the main concern was for the pushTxnQueue to be enabled after the call to leasePostApply, which it is because that's only gated by iAmTheLeaseHolder. The tests added in that PR only check to see if the queue is enabled, but don't test the contents.

Oh, gotcha, I was looking at the wrong part of the code, namely the one where the queue is disabled. No problem then!


Comments from Reviewable

@tbg
Copy link
Copy Markdown
Member

tbg commented Jul 20, 2017

- Before we would update the in-memory lease on a replica,
unlock it and then we would update the timestamp cache. This would
allow a replica to start processing requests for a lease without
first having taken into account reads served from the old lease
holder. We could see this by running:
make stress PKG=./pkg/sql/logictest
TESTS=TestParallel/subquery_retry_multinode STRESSFLAGS='-p 32'

This would fail because we would have duplicate values in a table
where inserts should have had monotonically increasing values.

- After this change the test passes. We also updated splitPostApply,
to send the current lease as the previous lease to leasePostApply to
prevent an update to the timestamp cache when a lease hasn't changed
hands.

Closes cockroachdb#16610
@m-schneider
Copy link
Copy Markdown
Contributor Author

Thanks!


Comments from Reviewable

@m-schneider m-schneider merged commit 406f7f7 into cockroachdb:master Jul 20, 2017
@m-schneider m-schneider deleted the leasechangefix branch July 20, 2017 15:22
@bdarnell
Copy link
Copy Markdown
Contributor

I think we want to cherry-pick this into 1.0.x. If we do it soon it can go in to 1.0.4 (the cutoff is today).


Review status: 1 of 2 files reviewed at latest revision, 2 unresolved discussions, all commit checks successful.


pkg/storage/store.go, line 1858 at r3 (raw file):

	// Invoke the leasePostApply method to ensure we properly initialize
	// the replica according to whether it holds the lease. This enables
	// the PushTxnQueue. Note that we pass in an right lease for prevLease so

This comment got a little muddled ("an right lease", and there's no longer a prevLease argument).


Comments from Reviewable

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants