Skip to content

kvserver: don't use caller's context for ProposalData#78174

Merged
craig[bot] merged 1 commit intocockroachdb:masterfrom
tbg:below-raft-no-caller-ctx
Mar 22, 2022
Merged

kvserver: don't use caller's context for ProposalData#78174
craig[bot] merged 1 commit intocockroachdb:masterfrom
tbg:below-raft-no-caller-ctx

Conversation

@tbg
Copy link
Copy Markdown
Member

@tbg tbg commented Mar 21, 2022

Fixes #75656.

Release note: None

@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

@tbg tbg force-pushed the below-raft-no-caller-ctx branch 3 times, most recently from d60e631 to 0610f35 Compare March 22, 2022 08:56
@tbg tbg marked this pull request as ready for review March 22, 2022 08:56
@tbg tbg requested a review from a team as a code owner March 22, 2022 08:56
@tbg tbg requested a review from a team March 22, 2022 09:02
Copy link
Copy Markdown
Contributor

@erikgrinaker erikgrinaker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks! Confirmed that tracing still works as expected.

It's a bit odd that the assertion is buried deep within some tracing method, but this is where we assign the command context and it makes sense to keep the patch small for backporting. We should consider restructuring this a bit later.

Previously, for local proposals, we were threading the caller's context
down into Raft. This has caused a few bugs related to context
cancellation sensitivity - the execution of a raft command should not be
impacted by whether or not the client's context has already finished.

This commit derives the proposal context from the Replica's annotated
context, using a tracing span derived from the client context. That
way, the derived context is not cancelable, but contains the same
tracing Span (thus providing similar utility) as before. Notably,
the trace span should also already contain the log tags present
in the client's context.

The tracing has coverage through (at least) a few tests that make
assertions about trace events from below-raft showing up in a trace
collected by the client, which I verified by omitting the Span:

```
TestRaftSSTableSideloadingProposal
TestReplicaAbandonProposal
TestReplicaRetryRaftProposal
TestReplicaBurstPendingCommandsAndRepropose
TestReplicaReproposalWithNewLeaseIndexError
TestFailureToProcessCommandClearsLocalResult
TestDBAddSSTable
```

See cockroachdb#75656.

Release note: None
@tbg tbg force-pushed the below-raft-no-caller-ctx branch from 0610f35 to 825a4ee Compare March 22, 2022 10:26
@tbg
Copy link
Copy Markdown
Member Author

tbg commented Mar 22, 2022

Had to rework this a bit (CI failed) - by creating the span above raft, we were starting to leak spans, and so there would have to have been more investigation and code. In the interest of time and keeping it small, I moved span creation below raft again. This is arguably the better change anyway, though now there is cmd.ctx and cmd.proposal.ctx and they're not the same; as you mentioned there's likely more work that we could do down the line.

PTAQL

Copy link
Copy Markdown
Contributor

@erikgrinaker erikgrinaker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this is fine for now.

@tbg
Copy link
Copy Markdown
Member Author

tbg commented Mar 22, 2022

Failure is TestLogic/fakedist/enums

bors r=[erikgrinaker,nvanbenschoten]

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Mar 22, 2022

Build failed (retrying...):

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Mar 22, 2022

Build succeeded:

@craig craig bot merged commit e0a4c51 into cockroachdb:master Mar 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

kvserver: use non-cancelable contexts in raft application

4 participants