Add initial checkpointing to the nullable walker.#35850
Add initial checkpointing to the nullable walker.#35850333fred merged 9 commits intodotnet:masterfrom
Conversation
| return operandType; | ||
| } | ||
|
|
||
| private void CheckpointWalkerThroughConversionGroup(BoundConversion conversionOpt, BoundExpression convertedNode) |
There was a problem hiding this comment.
I chose to add this as a separate method instead of putting it in RemoveConversion directly, as RemoveConversion is often used without then visiting the underlying expression. There are tradeoffs to both though, so there's preference on adding it to RemoveConversion, controlled by a default parameter, I can do that as well. #ByDesign
| // Currently, it will return incorrect info because it's just running analysis on the speculated | ||
| // part. | ||
| return NullableWalker.AnalyzeAndRewrite(Compilation, MemberSymbol as MethodSymbol, boundRoot, binder, diagnostics); | ||
| return NullableWalker.AnalyzeAndRewrite(Compilation, MemberSymbol as MethodSymbol, boundRoot, binder, diagnostics, checkpointMapOpt: null); |
There was a problem hiding this comment.
In a follow-up PR I will be asserting that a checkpoint map isn't passed to this version of the semantic model. #ByDesign
| var pooledDict = PooledDictionary<K, V>.GetInstance(); | ||
| foreach (var (key, value) in dictOpt) | ||
| { | ||
| pooledDict[key] = value; |
There was a problem hiding this comment.
pooledDict[key] = value [](start = 24, length = 23)
Please use .Add(key, value) to ensure there are no duplicates. #Resolved
| ArrayBuilder<(BoundReturnStatement, TypeWithAnnotations)> returnTypesOpt = null; | ||
| if (!_returnTypesOpt.IsDefault) | ||
| { | ||
| returnTypesOpt = ArrayBuilder<(BoundReturnStatement, TypeWithAnnotations)>.GetInstance(_returnTypesOpt.Length); |
There was a problem hiding this comment.
ArrayBuilder [](start = 37, length = 12)
new ArrayBuilder<...>() since it doesn't look like this will be freed. #Resolved
| Binder binder, | ||
| DiagnosticBag diagnostics) | ||
| DiagnosticBag diagnostics, | ||
| Dictionary<BoundNode, Checkpoint> checkpointMapOpt) |
There was a problem hiding this comment.
Dictionary<BoundNode, Checkpoint> checkpointMapOpt [](start = 12, length = 50)
Could this be returned as an ImmutableDictionary<,>? #Resolved
There was a problem hiding this comment.
I expect this method to be used both by the speculative semantic path and non-speculative path, and the speculative path won't need to actually do the checkpointing. I could change this to be a bool flag about whether to do the checkpointing and return null in that case if you'd prefer.
In reply to: 286218983 [](ancestors = 286218983)
There was a problem hiding this comment.
Changing to bool createMap, out ImmutableDictionary<,> map sounds reasonable.
In reply to: 286219930 [](ancestors = 286219930,286218983)
| walker._useMethodSignatureParameterTypes, | ||
| walker._symbol, | ||
| walker._returnTypesOpt?.ToImmutable() ?? default, | ||
| walker._methodGroupReceiverMapOpt?.ToImmutableDictionary(), |
There was a problem hiding this comment.
walker._methodGroupReceiverMapOpt?.ToImmutableDictionary(), [](start = 20, length = 59)
Can we share this ImmutableDictionary<,> with all checkpoints, until the original map changes? Same question for other immutable snapshots here. #Resolved
There was a problem hiding this comment.
If we're creating checkpoints at each node, then perhaps the actual NullableWalker fields should be immutable objects so the checkpoints can simply copy the fields by reference.
In reply to: 286220737 [](ancestors = 286220737)
There was a problem hiding this comment.
This will force us to create allocations for these new objects in the general straight-line case, which is a tradeoff we can make but I want to be sure we're deliberate about choosing to do so.
In reply to: 286227877 [](ancestors = 286227877,286220737)
| /// Contains a checkpoint for the NullableWalker at any given point of execution, used for restoring the walker to | ||
| /// a specific point for speculatively analyzing a piece of code that does not appear in the original tree. | ||
| /// </summary> | ||
| internal readonly struct Checkpoint |
There was a problem hiding this comment.
Checkpoint [](start = 33, length = 10)
Should this be named "Snapshot" perhaps? "Checkpoint" sounds like a point in time where a number of checks are performed rather than capturing state. #Resolved
There was a problem hiding this comment.
I don't know if it's the gamer in me, but in video games a "checkpoint" is the last state you go back to if you fail the level. I can use a different word if you really want, but checkpoint is correct for this as well.
In reply to: 286230409 [](ancestors = 286230409)
…sible, and add a caching strategy to the snapshotter to reduce garbage creation for every node. Addressed minor PR feedback items as well.
| /// Contains a checkpoint for the NullableWalker at any given point of execution, used for restoring the walker to | ||
| /// a specific point for speculatively analyzing a piece of code that does not appear in the original tree. | ||
| /// </summary> | ||
| internal readonly struct Snapshot |
There was a problem hiding this comment.
Snapshot [](start = 33, length = 8)
It feels like this structure can be significantly trimmed as we discussed offline. #Closed
| binder.Conversions, | ||
| this._variableState, | ||
| returnTypesOpt: default, | ||
| analyzedNullabilityMap, |
There was a problem hiding this comment.
analyzedNullabilityMap [](start = 20, length = 22)
Is the walker going to add to the map or to look for something in it? #Closed
There was a problem hiding this comment.
It will add to the map. This is how nullability info on the speculative code will be conveyed to the speculative model.
In reply to: 288720277 [](ancestors = 288720277)
| { | ||
| // The first slot always does not exist for the purposes of caching and can be ignored | ||
| // in the exit condition check | ||
| if (i != 0 && !walkerVariableBySlot[i].Exists) |
There was a problem hiding this comment.
i != 0 [](start = 28, length = 6)
Consider moving this logic out of the loop and starting at i = 1. #Closed
There was a problem hiding this comment.
I'm ripping out this caching logic entirely, so I will close these comments as won't fix as they're no longer relevant.
In reply to: 288751142 [](ancestors = 288751142)
| // in the exit condition check | ||
| if (i != 0 && !walkerVariableBySlot[i].Exists) | ||
| { | ||
| break; |
There was a problem hiding this comment.
break; [](start = 28, length = 6)
Can we use nextVariableSlot instead to figure out required capacity of the builder and to know how many items to copy with AddRange? #Closed
| int i; | ||
| for (i = 0; i < _lastVariableBySlot.Length; i++) | ||
| { | ||
| Debug.Assert(compareIdentifiers(_lastVariableBySlot[i], _walkerToSnapshot.variableBySlot[i])); |
There was a problem hiding this comment.
Debug.Assert(compareIdentifiers(_lastVariableBySlot[i], _walkerToSnapshot.variableBySlot[i])); [](start = 24, length = 94)
It looks like this code can be simplified if we assert that _lastVariableBySlot.Length == nextVariableSlot #Closed
There was a problem hiding this comment.
Assuming that we handle always empty 0's element outside of the loop.
In reply to: 288757319 [](ancestors = 288757319)
| Debug.Assert(_lastVariableSlot.Count == _walkerToSnapshot._variableSlot.Count); | ||
| foreach ((VariableIdentifier key, int value) in _lastVariableSlot) | ||
| { | ||
| if (!key.Exists) continue; |
There was a problem hiding this comment.
continue; [](start = 41, length = 9)
Is this code reachable? #Closed
There was a problem hiding this comment.
Yes, slot 0 never exists. However, this code is now gone.
In reply to: 288758660 [](ancestors = 288758660)
| #endif | ||
| } | ||
|
|
||
| internal void SnapshotWalker(BoundNode node) |
There was a problem hiding this comment.
SnapshotWalker [](start = 26, length = 14)
Is this function supposed to take a snapshot? The name is very confusing then. Consider renaming. #Closed
| _lastConditionalAccessSlot = lastConditionalAccessSlot; | ||
| } | ||
|
|
||
| internal static Snapshot SnapshotWalker(NullableWalker walker) => |
There was a problem hiding this comment.
SnapshotWalker [](start = 37, length = 14)
The name feels very confusing. #Closed
|
|
||
| internal static Snapshot SnapshotWalker(NullableWalker walker) => | ||
| new Snapshot( | ||
| walker._snapshotter.GetVariableState(walker.State), |
There was a problem hiding this comment.
walker.State [](start = 57, length = 12)
Wouldn't we want to clone the state? #Closed
| /// taking a snapshot of the NullableWalker in order to be as efficient | ||
| /// as possible when creating many snapshots. | ||
| /// </summary> | ||
| private sealed class Snapshotter |
There was a problem hiding this comment.
Snapshotter [](start = 29, length = 11)
It feels like we should test this machinery one way or another.
| /// Return statements and the result types from analyzing the returned expressions. Used when inferring lambda return type in MethodTypeInferrer. | ||
| /// </summary> | ||
| private readonly ArrayBuilder<(BoundReturnStatement, TypeWithAnnotations)> _returnTypesOpt; | ||
| private ImmutableArray<(BoundReturnStatement, TypeWithAnnotations)> _returnTypesOpt; |
There was a problem hiding this comment.
private ImmutableArray<(BoundReturnStatement, TypeWithAnnotations)> _returnTypesOpt; [](start = 8, length = 84)
I do not believe there is any advantage to change the type of this field to immutable array. #Closed
| /// (Could be replaced by _analyzedNullabilityMapOpt if that map is always available.) | ||
| /// </summary> | ||
| private PooledDictionary<BoundExpression, TypeWithState> _methodGroupReceiverMapOpt; | ||
| private ImmutableDictionary<BoundExpression, TypeWithState> _methodGroupReceiverMapOpt; |
There was a problem hiding this comment.
private ImmutableDictionary<BoundExpression, TypeWithState> _methodGroupReceiverMapOpt; [](start = 8, length = 87)
I do not believe there is any advantage in changing the type of this field. #Closed
| /// Placeholder locals, e.g. for objects being constructed. | ||
| /// </summary> | ||
| private PooledDictionary<object, PlaceholderLocal> _placeholderLocalsOpt; | ||
| private ImmutableDictionary<object, PlaceholderLocal> _placeholderLocalsOpt; |
There was a problem hiding this comment.
private ImmutableDictionary<object, PlaceholderLocal> _placeholderLocalsOpt; [](start = 8, length = 76)
I do not believe there is any advantage in changing the type of this field. #Closed
| ArrayBuilder<(BoundReturnStatement, TypeWithAnnotations)> returnTypesOpt, | ||
| VariableState initialState, | ||
| Dictionary<BoundExpression, (NullabilityInfo, TypeSymbol)> analyzedNullabilityMapOpt) | ||
| ImmutableArray<(BoundReturnStatement, TypeWithAnnotations)> returnTypesOpt, |
There was a problem hiding this comment.
ImmutableArray<(BoundReturnStatement, TypeWithAnnotations)> returnTypesOpt [](start = 12, length = 74)
What is the point in passing this array in? Would we ever use its content during analysis? #Closed
| foreach (var pair in initialState.VariableTypes) | ||
|
|
||
| var variableTypesBuilder = PooledDictionary<Symbol, TypeWithAnnotations>.GetInstance(); | ||
| foreach (var (key, value) in initialState.VariableTypes) |
There was a problem hiding this comment.
initialState.VariableTypes [](start = 45, length = 26)
Isn't this already an immutable dictionary? Can't we just use it as is? #Closed
| } | ||
| } | ||
|
|
||
| internal ImmutableDictionary<BoundNode, Snapshot> Snapshots => _snapshotMap.ToImmutableDictionary(); |
There was a problem hiding this comment.
_snapshotMap [](start = 75, length = 12)
Should we handle null here? #Closed
There was a problem hiding this comment.
No longer necessary: if there's a SnapshotManager, there is no null chance.
In reply to: 288779165 [](ancestors = 288779165)
| _currentSymbol = symbol.ContainingSymbol; | ||
| #endif | ||
| _walkerStates.SetItem(_currentWalkerSlot, stableState); | ||
| if (_initialSymbol.Equals(symbol)) |
There was a problem hiding this comment.
_initialSymbol.Equals(symbol) [](start = 24, length = 29)
Should be able to use reference equality here #Closed
| _currentSymbol = symbol.ContainingSymbol; | ||
| #endif | ||
| _walkerStates.SetItem(_currentWalkerSlot, stableState); | ||
| if (_initialSymbol.Equals(symbol)) |
There was a problem hiding this comment.
_initialSymbol.Equals(symbol) [](start = 24, length = 29)
I think we can get rid of _initialSymbol field. Here we can check if _currentWalkerSlot == 0 instead and it doesn't look like it adds much value elsewhere. #Closed
| if (_initialSymbol.Equals(symbol)) | ||
| { | ||
| Debug.Assert(_currentWalkerSlot == 0); | ||
| _currentWalkerSlot = -1; |
There was a problem hiding this comment.
_currentWalkerSlot = -1; [](start = 24, length = 24)
It feels like we would want to reset _currentSymbol to null as well. #Closed
There was a problem hiding this comment.
|
Done with review pass (iteration 6) #Closed |
| _currentSymbol = symbol; | ||
| #endif | ||
| _initialSymbol ??= symbol; | ||
| var previousSlot = _currentWalkerSlot; |
There was a problem hiding this comment.
previousSlot [](start = 24, length = 12)
Not used. #Resolved
…evert some changes in this PR to original state.
5a246c3 to
893239e
Compare
|
@AlekseyTs @cston addressed feedback. In reply to: 497871089 [](ancestors = 497871089) |
|
|
||
| internal int EnterNewWalker(Symbol symbol) | ||
| { | ||
| Debug.Assert(symbol is object); |
There was a problem hiding this comment.
@AlekseyTs unfortunately the assertion that this incoming Symbol is related to the current symbol proved to not be true in bad code scenarios like this one: https://github.com/dotnet/roslyn/blob/master/src/Compilers/CSharp/Test/Semantic/Semantics/LookupTests.cs#L1540-L1555.
| // will be called again from NullableWalker.ApplyConversion when the | ||
| // BoundLambda is converted to an anonymous function. | ||
| // https://github.com/dotnet/roslyn/issues/31752: Can we avoid generating extra | ||
| // diagnostics? And is this exponential when there are nested lambdas? |
There was a problem hiding this comment.
? [](start = 85, length = 1)
Dropped a "?" #Closed
|
|
||
| private void SnapshotWalkerThroughConversionGroup(BoundConversion conversionOpt, BoundExpression convertedNode) | ||
| { | ||
| var conversionGroup = conversionOpt?.ConversionGroupOpt; |
There was a problem hiding this comment.
var [](start = 12, length = 3)
Consider returning early if _snapshotBuilderOpt is null. #Closed
| return operandType; | ||
| } | ||
|
|
||
| private void SnapshotWalkerThroughConversionGroup(BoundConversion conversionOpt, BoundExpression convertedNode) |
There was a problem hiding this comment.
BoundConversion conversionOpt [](start = 58, length = 29)
All callers pass expr as BoundConversion. Consider changing this argument to BoundExpression and checking for BoundConversion here. (We're already performing that check inside the while.) #Closed
|
|
||
| public override BoundNode Visit(BoundNode node) | ||
| { | ||
| // Ensure that we always have a checkpoint for every BoundExpression in the map |
There was a problem hiding this comment.
checkpoint [](start = 48, length = 10)
"snapshot" #Closed
| } | ||
|
|
||
| /// <summary> | ||
| /// Contains the shared state state used to restore the walker at a specific point |
There was a problem hiding this comment.
state state [](start = 32, length = 11)
Typo #Closed
| private readonly struct Snapshot | ||
| { | ||
| internal readonly LocalState VariableState; | ||
| internal readonly int GlobalStateIndex; |
There was a problem hiding this comment.
Global [](start = 34, length = 6)
Consider using "Shared" rather than introducing a different term. #Closed
| foreach (var arm in node.SwitchArms) | ||
| { | ||
| SetState(!arm.Pattern.HasErrors && labelStateMap.TryGetValue(arm.Label, out var labelState) ? labelState.state : UnreachableState()); | ||
| // https://github.com/dotnet/roslyn/issues/35836 Is this where we want to take the checkpoint? |
There was a problem hiding this comment.
checkpoint [](start = 99, length = 10)
"snapshot" #Closed
| /// <summary> | ||
| /// The int key corresponds to <see cref="Snapshot.GlobalStateIndex"/>. | ||
| /// </summary> | ||
| private readonly ImmutableArray<SharedWalkerState> _walkerGlobalStates; |
There was a problem hiding this comment.
ImmutableArray _walkerGlobalStates [](start = 29, length = 53)
Is this array necessary? If SharedWalkerState were a class, could we use a reference to SharedWalkerState in Snapshot rather than an index into this array? #WontFix
There was a problem hiding this comment.
We could potentially do that, but I'm going to leave this as is. I think making SharedWalkerState a class would probably introduce more garbage than it would save.
In reply to: 289925758 [](ancestors = 289925758)
There was a problem hiding this comment.
If the concern is allocations, the fields of SharedWalkerState could be copied into Snapshot.
In reply to: 289940273 [](ancestors = 289940273,289925758)
| /// closest position before a given position. | ||
| /// </summary> | ||
| private sealed class DescendingIntComparer : IComparer<int> | ||
| { |
There was a problem hiding this comment.
{ [](start = 12, length = 1)
Consider adding a singleton instance. #Closed
|
@AlekseyTs @cston addressed feedback. |
| /// <summary> | ||
| /// The int key corresponds to <see cref="Snapshot.SharedStateIndex"/>. | ||
| /// </summary> | ||
| private readonly ImmutableArray<SharedWalkerState> _walkerGlobalStates; |
There was a problem hiding this comment.
Global [](start = 70, length = 6)
"Shared"?
| return null; | ||
| } | ||
|
|
||
| var globalState = _walkerGlobalStates[incrementalSnapshot.SharedStateIndex]; |
There was a problem hiding this comment.
globalState [](start = 20, length = 11)
"sharedState"?
| internal readonly LocalState VariableState; | ||
| internal readonly int SharedStateIndex; | ||
|
|
||
| internal Snapshot(LocalState variableState, int globalStateIndex) |
There was a problem hiding this comment.
global [](start = 60, length = 6)
"shared"
|
@cston I'll address those comments in the next followup PR. |
This adds the initial code for checkpointing the nullable walker, and defines the structure of a checkpoint. This PR does not do anything with the checkpoints yet: MemberSemanticModel gets the checkpoints, and then just throws them away without saving them. The intention is that it will save the checkpoints and use them in the speculative model in a follow up PR. @dotnet/roslyn-compiler @AlekseyTs @gafter for review.