Cleanup our temp-storage (memory mapped file) code. #73115
Cleanup our temp-storage (memory mapped file) code. #73115CyrusNajmabadi merged 80 commits intodotnet:mainfrom
Conversation
| @@ -110,7 +110,7 @@ protected override PortableExecutableReference WithPropertiesImpl(MetadataRefere | |||
| private string GetDebuggerDisplay() | |||
There was a problem hiding this comment.
I recommend reviewing this PR in a particular order. First start with the ITemporaryStorageServiceInternal change.
| Stream ReadFromTemporaryStorageService(TemporaryStorageIdentifier storageIdentifier, CancellationToken cancellationToken); | ||
|
|
||
| ITemporaryTextStorageInternal CreateTemporaryTextStorage(); | ||
| } |
There was a problem hiding this comment.
here's the crux of the change. The prior API (CreateTemporaryStorageStream) would effectively give you an instance of an object you could read or write to. But that object was super strange. For example, you couldn't actually read from this object (since nothing had written to it). And, if you wrote to it, you could only write to it once.
Furthermore, that value returned happened to implement other APIs as well taht would allow passing information through a backdoor to help inform OOP serialization. Specifically, what the mmf information was for the written data so we could send that data over to OOP to let it hydrate the info on its side.
The API changes heavily now. First off, when you want to write some stream to a mmf, you just say WriteToTemporaryStorage passing in the data you actually want to write up front. This gives you back a 'handle' representing that data. The handle allows getting the mmf data for cases like OOP. It also supports releasing that written data when no longer needed.
Interestingly enough though, that 'releasing' behavior is only used in exactly one place in roslyn. And that place is unrelated to SourceTexts or References, and is unrelated to communicating with OOP. It's literally where we dump the enormous compiler-command-line string we build that we don't want actually staying in memory. All the OOP cases, and all the SourceText/Reference mmf dumping never releases the data from the memory mapped files. This is likely a leak in the traditional sense. But one that is so slow that it is rarely noticed in practice (no one has ever mentioned it up till now).
Because the data is never released, the protocol with the server is simple, as the server can just depend on this data stayign alive.
--
The next method that is added is the ReadFromTemporaryStorageService method. This takes the information produced by the WriteToTemporaryStorage, using it to erad from teh MMF and supply the data in a stream.
--
Once this has been read, you cna jump back to the start of the review.
| public IReadOnlyList<ITemporaryStreamStorageInternal> GetStorages() | ||
| => _provider.GetStorages(this.FilePath, _timestamp.Value); | ||
| public IReadOnlyList<TemporaryStorageIdentifier> GetStorageIdentifiers() | ||
| => _provider.GetStorageIdentifiers(this.FilePath, _timestamp.Value); |
There was a problem hiding this comment.
the old name was just super confusing. what does 'GetStorages' even mean? Also, what it returned was also weird. It returned the old object htat you could read/write to (though nothing ever read/wrote to it). So what was it used for? Well, callers would cast this type to another type to try to get the MMF info to stream to OOP.
All this gets simpler by havnig a real type to represetnt that data, and having tehse helpers just return instances of those types alone. Not the actual storage object.
...sualStudio/Core/Def/ProjectSystem/MetadataReferences/VisualStudioMetadataReferenceManager.cs
Outdated
Show resolved
Hide resolved
| /// to temporary storage and get a handle to it. Use <see cref="ReadFromTemporaryStorage"/> to read the data back in | ||
| /// any process. | ||
| /// </summary> | ||
| internal interface ITemporaryStorageHandle |
src/Workspaces/Core/Portable/Workspace/Host/TemporaryStorage/TemporaryStorageHandle.cs
Outdated
Show resolved
Hide resolved
...rkspaces/Core/Portable/Workspace/Solution/SolutionCompilationState.SkeletonReferenceCache.cs
Show resolved
Hide resolved
src/Workspaces/Core/Portable/Workspace/ProjectSystem/ProjectSystemProjectOptionsProcessor.cs
Outdated
Show resolved
Hide resolved
|
@jasonmalinowski For review when you get back. |
Explanation given in https://github.com/dotnet/roslyn/pull/73115/files#r1573125392 for the motivation and main thrust of the change here. Effectively, it's that the entire shape and interaction pattern of dealing with MMFs is fairly busted. Akin to having an API wrongly expose itself as an IEnumerator instead of as an IEnumerable. Except here, the API exposes itself as a single-write or single-read entity, and decides what to do depending on which you call first. This PR separates out those concepts.
PR1: https://devdiv.visualstudio.com/DefaultCollection/DevDiv/_build/results?buildId=9452884&view=results
PR2: https://devdiv.visualstudio.com/DefaultCollection/DevDiv/_build/results?buildId=9452885&view=results