Skip to content

kvserver: (partially) deflake transfer-leases/drain-other-node#88205

Merged
craig[bot] merged 1 commit intocockroachdb:masterfrom
irfansharif:220919.drain-leases
Sep 21, 2022
Merged

kvserver: (partially) deflake transfer-leases/drain-other-node#88205
craig[bot] merged 1 commit intocockroachdb:masterfrom
irfansharif:220919.drain-leases

Conversation

@irfansharif
Copy link
Copy Markdown
Contributor

@irfansharif irfansharif commented Sep 20, 2022

In #85629 we changed our lease transfer protocol to only ever transfer expiration-based leases, and have recipients later upgrade them to the more efficient epoch based ones. This was done to limit the effects of ill-advised lease transfers since the incoming leaseholder would need to recognize itself as such within a few seconds -- so we wanted this upgrade happen after having received the lease.

In #83261 however we noticed that the upgrade was not immediate -- we were waiting until the current lease's expiration was within its renewal duration -- 4.5s. When the lease was eventually renewed the upgrade did happen, but it was not immediate. We fix this here and remove the manual clock advancing the supporting test had that masked this issue. It now demonstrates that we're no longer relying on upgrades happen as part of the (slow) renewal process.

Release note: None

@irfansharif irfansharif requested a review from nvb September 20, 2022 02:29
@irfansharif irfansharif requested a review from a team as a code owner September 20, 2022 02:29
@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

Copy link
Copy Markdown
Contributor

@nvb nvb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 3 of 3 files at r1, all commit messages.
Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @irfansharif)

In cockroachdb#85629 we changed our lease transfer protocol to only ever
transfer expiration-based leases, and have recipients later upgrade
them to the more efficient epoch based ones. This was done to limit
the effects of ill-advised lease transfers since the incoming
leaseholder would need to recognize itself as such within a few seconds
-- so we wanted this upgrade happen after having received
the lease.

In cockroachdb#83261 however we noticed that the upgrade was not immediate -- we
were waiting until the current lease's expiration was within its renewal
duration -- 4.5s. When the lease was eventually renewed the upgrade did
happen, but it was not immediate. We fix this here and remove the manual
clock advancing the supporting test had that masked this issue. It now
demonstrates that we're no longer relying on upgrades happen as part of
the (slow) renewal process.

Release note: None
Copy link
Copy Markdown
Contributor Author

@irfansharif irfansharif left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Had to update some other tests to explicitly wait for lease upgrades to happen, lest they run into errors around proposals being issues under different leases. Will wait on green CI to bors.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @nvanbenschoten)

@irfansharif
Copy link
Copy Markdown
Contributor Author

CI failure repros in the commit earlier.

bors r+

Copy link
Copy Markdown
Contributor

@nvb nvb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 2 of 2 files at r2, all commit messages.
Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @irfansharif)

@irfansharif
Copy link
Copy Markdown
Contributor Author

CI failure is #88051.

bors r+

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Sep 20, 2022

Already running a review

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Sep 20, 2022

Build failed (retrying...):

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Sep 21, 2022

Build succeeded:

@craig craig bot merged commit a25a8b0 into cockroachdb:master Sep 21, 2022
@irfansharif irfansharif deleted the 220919.drain-leases branch September 21, 2022 13:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants