Fix data race in compiler when resolver provides unlinked descriptor proto#103
Merged
Fix data race in compiler when resolver provides unlinked descriptor proto#103
Conversation
Closed
pkwarren
reviewed
Mar 2, 2023
Member
pkwarren
left a comment
There was a problem hiding this comment.
A bit concerned with the complexity around clone and also with any possible missed mutations in the checks to determine if we need to clone the parse result or descriptor proto.
pkwarren
approved these changes
Mar 2, 2023
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
From looking at the data race errors in bug report #102, this can happen only if a
*descriptorpb.FileDescriptorProtoorparser.Resultis provided as a search result and is shared across parallel compiler operations.So, luckily, that means many (most?) usages of the compiler are safe, including the way that
bufuses it.This looks to have always been the case. But I bet the user described this as a new failure because their tests previously didn't all share a search result for
"google/protobuf/descriptor.proto". But as of #97, all compile operations will implicitly use a search result for this file. So I suspect the reporter's tests use a resolver that returns a proto for this file and previously only one of the tests actually used it (i.e. only one test actually imports that file).This is a non-trivial change mainly because I was worried about the performance impact of always cloning values, even if it's not necessary. So the strategy here is to only make a defensive clone of the descriptor proto (or the parser result, which contains a potentially mutable descriptor proto) when necessary. How do we know if it's needed? If it has any type names that are not fully-qualified (field extendees, field types, method input and output types), if it has uninterpreted options, or if it has no source code info but needs it. Under these conditions, the compiler will mutate the proto (to make type names fully-qualified, to interpret options, to add source code info).
Edit: The code now always clones descriptor protos and parser results (which may contain a descriptor proto). That way there's no worry of the mutation logic getting out of sync with the logic that decides whether or not to clone.
The most complicated bit here is actually cloning a parser result, and associated tests, because it has an index that maps elements in the descriptor proto to their corresponding AST nodes. So we have to rebuild that index so it matches the new (cloned) elements in the descriptor proto.
The test added to
compiler_test.gosuccessfully reproduces the data race that was reported and is fixed with all of the other changes in this branch.Fixes #102.