Skip to content

kv: remove broken attempt to reject lease acquisitions on draining nodes#87885

Merged
craig[bot] merged 1 commit intocockroachdb:masterfrom
nvb:nvanbenschoten/leaseReqDraining2
Sep 21, 2022
Merged

kv: remove broken attempt to reject lease acquisitions on draining nodes#87885
craig[bot] merged 1 commit intocockroachdb:masterfrom
nvb:nvanbenschoten/leaseReqDraining2

Conversation

@nvb
Copy link
Copy Markdown
Contributor

@nvb nvb commented Sep 12, 2022

Related to #83261.

This commit removes "protection" that avoided lease acquisitions on draining nodes. This protection had already been effectively disabled by acc1ad1, which allowed Raft leaders to bypass the check. As the comment here (added in 5ffaa9e) explained, Raft followers are already unable to acquire the lease. If leaders bypass the check and follower (and candidates) don't need it, the check is useless, so we remove it.

The commit also removes TestReplicaDrainLease, which was supposed to test this behavior. We remove the test not because it started failing after the change, but because it did not. It must not have been testing anything real anymore after acc1ad1.

Release justification: low risk change related to release blocker.

Release note: None

@nvb nvb requested a review from a team as a code owner September 12, 2022 23:10
@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

@aayushshah15
Copy link
Copy Markdown
Contributor

pkg/kv/kvserver/replica_range_lease.go line 824 at r1 (raw file):

			)
		}
		log.Info(ctx, "trying to take the lease while we're draining since we're the raft leader")

I thought that this log line was useful in cases where draining nodes spuriously re-acquire leases. Do you think we can still keep this log line around? Do you think there's a better way to have this observability?

@nvb nvb force-pushed the nvanbenschoten/leaseReqDraining2 branch from ef2d7e9 to fafb80e Compare September 20, 2022 16:49
Copy link
Copy Markdown
Contributor Author

@nvb nvb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @aayushshah15)


pkg/kv/kvserver/replica_range_lease.go line 824 at r1 (raw file):

Previously, aayushshah15 (Aayush Shah) wrote…

I thought that this log line was useful in cases where draining nodes spuriously re-acquire leases. Do you think we can still keep this log line around? Do you think there's a better way to have this observability?

Good point. I unified this with the below-Raft lease acquisition logging.

// 3. verbose logging is enabled
epochLeaseChangingHands := newLease.Type() == roachpb.LeaseEpoch && leaseChangingHands
if epochLeaseChangingHands || r.store.IsDraining() || log.V(1) {
log.VEventf(ctx, 1, "new range lease %s following %s", newLease, prevLease)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be a log.Infof instead?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. This logic was slowly adapted by c3dd472 and c3dd472. I think the change I made strikes the desired balance of verbosity, but PTAL.

Related to cockroachdb#83261.

This commit removes "protection" that avoided lease acquisitions on
draining nodes. This protection had already been effectively disabled by
acc1ad1, which allowed Raft leaders to bypass the check. As the comment
here (added in 5ffaa9e) explained, Raft followers are already unable to
acquire the lease. If leaders bypass the check and follower (and
candidates) don't need it, the check is useless, so we remove it.

The commit also removes `TestReplicaDrainLease`, which was supposed to
test this behavior. We remove the test not because it started failing
after the change, but because it did not. It must not have been testing
anything real anymore after acc1ad1.

Release justification: low risk change related to release blocker.

Release note: None
@nvb nvb force-pushed the nvanbenschoten/leaseReqDraining2 branch from fafb80e to d89a7e4 Compare September 20, 2022 17:16
@nvb
Copy link
Copy Markdown
Contributor Author

nvb commented Sep 20, 2022

TFTR!

bors r+

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Sep 20, 2022

Build failed (retrying...):

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Sep 21, 2022

Build succeeded:

@craig craig bot merged commit a25a8b0 into cockroachdb:master Sep 21, 2022
@nvb nvb deleted the nvanbenschoten/leaseReqDraining2 branch September 21, 2022 19:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants