Background and Motivation
The Source Generator V2 APIs introduce the concept of "incremental generators". These generators describe their work as a pipeline of transformations and filters, which the generator driver can incrementally run steps of a generator. As part of supporting this, the API allows developers to provide custom comparers to enable using their own custom types between stages. For complex source generators that carry a lot of state, it is desirable to validate that the incrementality for various steps is preserved (a given step in the transform will result in the same value as a prior run, so later results based on this result are pulled from the cache).
This proposal proposes APIs to enable tracking this information without requiring incremental generators to carry state (which is undesired by the Roslyn team to my understanding).
Approved API
namespace Microsoft.CodeAnalysis
{
public static class IncrementalValueProviderExtensions
{
+ public static IncrementalValueProvider<TSource> WithTrackingName<TSource>(this IncrementalValueProvider<TSource> source, string name);
+ public static IncrementalValuesProvider<TSource> WithTrackingName<TSource>(this IncrementalValuesProvider<TSource> source, string name);
}
public readonly struct GeneratorDriverOptions
{
+ public readonly bool TrackIncrementalGeneratorSteps;
+ public GeneratorDriverOptions(IncrementalGeneratorOutputKind disabledOutputs, bool trackIncrementalGeneratorSteps);
}
public class GeneratorRunResult
{
+ public ImmutableDictionary<string, ImmutableArray<IncrementalGeneratorRunStep>> TrackedSteps { get; }
+ public ImmutableDictionary<string, ImmutableArray<IncrementalGeneratorRunStep>> TrackedOutputSteps { get; }
}
+ public static class WellKnownGeneratorInputs
+ {
+ public const string Compilation = nameof(Compilation);
+
+ public const string MetadataReferences = nameof(MetadataReferences);
+
+ public const string AdditionalTexts = nameof(AdditionalTexts);
+
+ public const string ParseOptions = nameof(ParseOptions);
+
+ public const string AnalyzerConfigOptions = nameof(AnalyzerConfigOptions);
+
+ public const string SyntaxTrees = nameof(SyntaxTrees);
+ }
+ public static class WellKnownGeneratorOutputs
+ {
+ public const string SourceOutput = nameof(SourceOutput);
+
+ public const string ImplementationSourceOutput = nameof(ImplementationSourceOutput);
+ }
+ public class IncrementalGeneratorRunStep
+ {
+ public string? Name { get; }
+ public ImmutableArray<(IncrementalGeneratorRunStep Source, int OutputIndex)> Inputs { get; }
+ public ImmutableArray<(object Value, IncrementalStepRunReason Reason)> Outputs { get; }
+ public TimeSpan ElapsedTime { get; }
+ }
+ public enum IncrementalStepRunReason
+ {
+ New,
+ Modified,
+ Unchanged,
+ Cached,
+ Removed
+ }
}Proposed API (Updated based on August 25, 2021 API Review)
Details
namespace Microsoft.CodeAnalysis
{
public static class IncrementalValueProviderExtensions
{
+ public static IncrementalValueProvider<TSource> WithTrackingName<TSource>(this IncrementalValueProvider<TSource> source, string name);
+ public static IncrementalValuesProvider<TSource> WithTrackingName<TSource>(this IncrementalValuesProvider<TSource> source, string name);
}
}
namespace Microsoft.CodeAnalysis
{
public readonly struct GeneratorDriverOptions
{
public readonly IncrementalGeneratorOutputKind DisabledOutputs;
+ public readonly bool TrackIncrementalGeneratorSteps;
public GeneratorDriverOptions(IncrementalGeneratorOutputKind disabledOutputs);
+ public GeneratorDriverOptions(IncrementalGeneratorOutputKind disabledOutputs, bool trackIncrementalGeneratorSteps);
}
public class GeneratorRunResult
{
+ public ImmutableDictionary<string, ImmutableArray<IncrementalGeneratorRunStep>> TrackedSteps { get; }
+ public ImmutableDictionary<string, ImmutableArray<IncrementalGeneratorRunStep>> TrackedOutputSteps { get; }
}
+ public class IncrementalGeneratorRunStep
+ {
+ public const string CompilationStep = "Compilation";
+ public const string ParseOptionsStep = "ParseOptions";
+ public const string AdditionalTextsStep = "AdditionalTexts";
+ public const string SyntaxTreesStep = "SyntaxTrees";
+ public const string AnalyzerConfigOptionsStep = "AnalyzerConfigOptions";
+ public const string MetadataReferencesStep = "MetadataReferences";
+ public const string SourceOutputStep = "SourceOutput";
+ public string? Name { get; }
+ public ImmutableArray<(IncrementalGeneratorRunStep Source, int OutputIndex)> Inputs { get; }
+ public ImmutableArray<(object Value, IncrementalStepOutputStatus Status)> Outputs { get; }
+ public TimeSpan ElapsedTime { get; }
+ }
+ public enum IncrementalStepOutputStatus
+ {
+ New,
+ Modified,
+ Unchanged,
+ Cached,
+ Removed
+ }
}
}Generators created with a GeneratorDriverOptions instance with TrackIncrementalGeneratorSteps=true would record the results of each of the steps of the various incremental generators run by the driver.
Open Question: Should the recorded steps include unnamed steps and named steps, or just named steps?
Usage Examples
class MyGenerator : IIncrementalGenerator
{
...
}
...
GeneratorDriver driver = CSharpGeneratorDriver.Create(new [] { new MyGenerator() }, null, null, null, new GeneratorDriverOptions(IncrementalGeneratorOutputKind.None, true));
Compilation comp1 = ...;
// Run the generators once to set up the baseline cached results.
driver = driver.RunGenerators(comp1);
Compilation comp2 = comp1.AddSyntaxTrees(...);
driver.RunGenerators(comp2);
GeneratorRunResult steps = driver.GetRunResults().Results[0];
Assert.Equal(IncrementalGeneratorRunStepState.InputModifiedOutputUnchanged, steps.ExecutedSteps["Calculate State"][0].Outputs[0].State);
Assert.Equal(IncrementalGeneratorRunStepState.InputUnchangedCachedOutputUsed, steps.ExecutedSteps["Generate Code"][0].Outputs[0].State);Alternative Designs
The step info could instead be designed as a linked list of step status through the generator chain instead of an array with no well-defined ordering. This design would make it easier to associate two steps together, but provides difficulties around excluding unnamed steps. Additionally, this design may have to "leak" implementation details about how the steps are chained together to efficiently construct the list.
Risks
As the rules around when a generator considers a change a "modify" change vs a "remove/add" change are not well-defined, the determination around how they are codified in this API may risk causing breaking changes in the future if the rules change.
Background and Motivation
The Source Generator V2 APIs introduce the concept of "incremental generators". These generators describe their work as a pipeline of transformations and filters, which the generator driver can incrementally run steps of a generator. As part of supporting this, the API allows developers to provide custom comparers to enable using their own custom types between stages. For complex source generators that carry a lot of state, it is desirable to validate that the incrementality for various steps is preserved (a given step in the transform will result in the same value as a prior run, so later results based on this result are pulled from the cache).
This proposal proposes APIs to enable tracking this information without requiring incremental generators to carry state (which is undesired by the Roslyn team to my understanding).
Approved API
namespace Microsoft.CodeAnalysis { public static class IncrementalValueProviderExtensions { + public static IncrementalValueProvider<TSource> WithTrackingName<TSource>(this IncrementalValueProvider<TSource> source, string name); + public static IncrementalValuesProvider<TSource> WithTrackingName<TSource>(this IncrementalValuesProvider<TSource> source, string name); } public readonly struct GeneratorDriverOptions { + public readonly bool TrackIncrementalGeneratorSteps; + public GeneratorDriverOptions(IncrementalGeneratorOutputKind disabledOutputs, bool trackIncrementalGeneratorSteps); } public class GeneratorRunResult { + public ImmutableDictionary<string, ImmutableArray<IncrementalGeneratorRunStep>> TrackedSteps { get; } + public ImmutableDictionary<string, ImmutableArray<IncrementalGeneratorRunStep>> TrackedOutputSteps { get; } } + public static class WellKnownGeneratorInputs + { + public const string Compilation = nameof(Compilation); + + public const string MetadataReferences = nameof(MetadataReferences); + + public const string AdditionalTexts = nameof(AdditionalTexts); + + public const string ParseOptions = nameof(ParseOptions); + + public const string AnalyzerConfigOptions = nameof(AnalyzerConfigOptions); + + public const string SyntaxTrees = nameof(SyntaxTrees); + } + public static class WellKnownGeneratorOutputs + { + public const string SourceOutput = nameof(SourceOutput); + + public const string ImplementationSourceOutput = nameof(ImplementationSourceOutput); + } + public class IncrementalGeneratorRunStep + { + public string? Name { get; } + public ImmutableArray<(IncrementalGeneratorRunStep Source, int OutputIndex)> Inputs { get; } + public ImmutableArray<(object Value, IncrementalStepRunReason Reason)> Outputs { get; } + public TimeSpan ElapsedTime { get; } + } + public enum IncrementalStepRunReason + { + New, + Modified, + Unchanged, + Cached, + Removed + } }Proposed API (Updated based on August 25, 2021 API Review)
Details
namespace Microsoft.CodeAnalysis { public static class IncrementalValueProviderExtensions { + public static IncrementalValueProvider<TSource> WithTrackingName<TSource>(this IncrementalValueProvider<TSource> source, string name); + public static IncrementalValuesProvider<TSource> WithTrackingName<TSource>(this IncrementalValuesProvider<TSource> source, string name); } } namespace Microsoft.CodeAnalysis { public readonly struct GeneratorDriverOptions { public readonly IncrementalGeneratorOutputKind DisabledOutputs; + public readonly bool TrackIncrementalGeneratorSteps; public GeneratorDriverOptions(IncrementalGeneratorOutputKind disabledOutputs); + public GeneratorDriverOptions(IncrementalGeneratorOutputKind disabledOutputs, bool trackIncrementalGeneratorSteps); } public class GeneratorRunResult { + public ImmutableDictionary<string, ImmutableArray<IncrementalGeneratorRunStep>> TrackedSteps { get; } + public ImmutableDictionary<string, ImmutableArray<IncrementalGeneratorRunStep>> TrackedOutputSteps { get; } } + public class IncrementalGeneratorRunStep + { + public const string CompilationStep = "Compilation"; + public const string ParseOptionsStep = "ParseOptions"; + public const string AdditionalTextsStep = "AdditionalTexts"; + public const string SyntaxTreesStep = "SyntaxTrees"; + public const string AnalyzerConfigOptionsStep = "AnalyzerConfigOptions"; + public const string MetadataReferencesStep = "MetadataReferences"; + public const string SourceOutputStep = "SourceOutput"; + public string? Name { get; } + public ImmutableArray<(IncrementalGeneratorRunStep Source, int OutputIndex)> Inputs { get; } + public ImmutableArray<(object Value, IncrementalStepOutputStatus Status)> Outputs { get; } + public TimeSpan ElapsedTime { get; } + } + public enum IncrementalStepOutputStatus + { + New, + Modified, + Unchanged, + Cached, + Removed + } } }Generators created with a
GeneratorDriverOptionsinstance withTrackIncrementalGeneratorSteps=truewould record the results of each of the steps of the various incremental generators run by the driver.Open Question: Should the recorded steps include unnamed steps and named steps, or just named steps?
Usage Examples
Alternative Designs
The step info could instead be designed as a linked list of step status through the generator chain instead of an array with no well-defined ordering. This design would make it easier to associate two steps together, but provides difficulties around excluding unnamed steps. Additionally, this design may have to "leak" implementation details about how the steps are chained together to efficiently construct the list.
Risks
As the rules around when a generator considers a change a "modify" change vs a "remove/add" change are not well-defined, the determination around how they are codified in this API may risk causing breaking changes in the future if the rules change.