Visual Studio has recently added [AppliesToProject] support for IAttachedCollectionSourceProvider providers, which can be used to prevent unneeded dlls. Roslyn should use this for all their implementations inside Microsoft.VisualStudio.LanguageServices.SolutionExplorer.dll to avoid unneeded loads of this dll in projects/solutions where it is not applicable, for example:

    [Export(typeof(IAttachedCollectionSourceProvider))]
    [Name("AnalyzerItemsProvider")]
    [Order]
    [AppliesToProject("(CSharp | VisualBasic) && !CPS"] // Legacy C# or VB projects

and:

    [Export(typeof(IAttachedCollectionSourceProvider))]
    [Name(nameof(CpsDiagnosticItemProvider))]
    [Order]
    [AppliesToProject("(CSharp | VisualBasic) && CPS"] // CPS-based C# or VB projects

Note: Similar to UI contexts for commands/packages this only controls whether your implementation is loaded for a given solution. You will still be called back for projects that do not contain these capabilities and you will need to continue to handle this.