Skip to content

Apply a solution filter#4899

Merged
Forgind merged 28 commits intodotnet:masterfrom
Forgind:slnf-filter
Jan 9, 2020
Merged

Apply a solution filter#4899
Forgind merged 28 commits intodotnet:masterfrom
Forgind:slnf-filter

Conversation

@Forgind
Copy link
Contributor

@Forgind Forgind commented Nov 7, 2019

This fixes the fact that, when building, we always build a complete solution even if only a small part of it is really necessary. One workaround is to have a stripped-down .sln file, but .slnf has a simpler format and less duplicated information. This creates support for .slnf files in MSBuild.

Fixes #4097.

Filter files currently look like this:

  "solution": {
    "path": "C:\\Users\\namytelk\\Documents\\GitHub\\msbuild\\MSBuild.Dev.sln",
    "projects": [
      "src\\Build\\Microsoft.Build.csproj",
      "src\\Framework\\Microsoft.Build.Framework.csproj",
      "src\\MSBuild\\MSBuild.csproj",
      "src\\Tasks.UnitTests\\Microsoft.Build.Tasks.UnitTests.csproj"
    ]
  }
}

Documentation can be found here.

Copy link
Member

@rainersigwald rainersigwald left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

// If the project is an etp project then parse the etp project file
// to get the projects contained in it.
if (IsEtpProjectFile(proj.RelativePath))
if (slnFilter == null || slnFilter.Contains(proj.RelativePath))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this valid? I would have expected that we'd need to understand all projects in the solution for configuration purposes, even if they're not valid entry points.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When you make a solution filter file, it automatically checks for dependencies, so my understanding is that you will either have all the necessary projects in the filter file, or it should fail.

Copy link
Member

@rainersigwald rainersigwald left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approach looks good, I think. Needs tests, and probably better error handling.

{
JsonElement solution = text.RootElement.GetProperty("solution");
_solutionFile = solution.GetProperty("path").GetString();
_solutionFilter = new HashSet<string>(NativeMethodsShared.IsLinux ? StringComparer.Ordinal : StringComparer.OrdinalIgnoreCase);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where did we land on this case insensitivity/how does VS currently behave?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I modify a .sln file by changing case, VS handles it fine. If I modify a .slnf file to change case, it has trouble finding it. Since .sln files have existed much longer than .slnf files, I wouldn't be surprised if .slnf files change at some point, so I would prefer to stick with ignoring case.


#region Methods

internal bool IsFiltered(string projectFile)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
internal bool IsFiltered(string projectFile)
internal bool FilterIncludesProject(string projectFile)

?

{
return _solutionFilter == null || _solutionFilter.Contains(projectFile);
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think this method should be more intelligent? In particular, I was wondering if we should check if there's a mismatch between full and relative paths. One extra complication for this is that paths in the .slnf are, by default, relative from the .sln file, which might include some ..s that would have to be simplified out.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand the question. Can you clarify? I would expect the path constraints on projects in the solution to match the slnf; is that not true?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, both the .sln and .slnf use relative paths from the .sln, so there's no problem. As you noted at one point, however, .slnfs are easy to modify, so this would make it resistant to adding an absolute path into the .slnf.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd rather reject such an slnf than try to tolerate it (unless VS does)

public string GetDefaultConfigurationName() { throw null; }
public string GetDefaultPlatformName() { throw null; }
public static Microsoft.Build.Construction.SolutionFile Parse(string solutionFile) { throw null; }
public bool ProjectShouldBuild(string projectFile) { throw null; }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this need to be public? What's the expected API consumer going to do with it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It needs to be visible to SolutionFile_Tests.cs, and given how long it took to resolve the previous IVT change, I'm somewhat reluctant to make it internal and add more IVT changes.

{
if (FrameworkLocationHelper.PathToDotNetFrameworkV35 == null)
{
// ".NET Framework 3.5 is required to be installed for this test, but it is not installed.");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does this new test depend on .NET 3.5?

_solutionFile = value;
if (FileUtilities.IsSolutionFilterFilename(value))
{

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: delete blank line

///
/// </summary>
[Fact]
public void InvalidSolutionFilters()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since you have several test cases here, use a Theory, and give the different cases names.

string filterFileContents = @"
{
'solution': {
'path': 'C:\\notAPath\\MSBuild.Dev.sln',
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assert that this file doesn't exist since it's a valid path.


if (_solutionFilter != null)
{
HashSet<string> projectNamesInOrder = new HashSet<string>(_projectsInOrder.Count);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this have the same comparer case sensitivity as the solution filter?

int visualStudioVersion = 0;
if (FileUtilities.IsSolutionFilterFilename(projectFile))
{
solutionVersion = 16;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might age badly if there's a 17 that supports slnf and some new thing. Can you get it from the referenced .sln?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. My rationale is that it wouldn't make any difference now, and this way avoids file i/o (slow) and a string search. Would you be ok with it if I just put a comment like // TODO: parse file to get accurate solution version?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's no guarantee that we will rediscover it in time to prevent the future problems, and no way to ensure the TODO is done at a specific time, so it's usually a good idea to address this sort of thing as soon as it's discovered.

However, is this even relevant? Is the change down in line 2074 all we need here, and we just need to get to it? If that's the case can we make the logic say that more clearly?

<value>Path: {0} exceeds the OS max path limit. The fully qualified file name must be less than {1} characters.</value>
<comment></comment>
</data>
<data name="JsonParsingError" UESanitized="false" Visibility="Public">
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Convention in shared resources is to start with the Shared. prefix.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are a lot without that prefix (see SolutionParseVersionMismatchError or UnrecognizedSolutionComment for instance). Do you want me to change those as well?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, I don't see any examples from SolutionFile with the Shared. prefix. Maybe there's an exception for some reason?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wouldn't go back and change the others, but I'd expect anything in this file (compiled into multiple assemblies) to have the prefix. I don't know why the existing solution ones don't, that's funky.

<value>Path: {0} exceeds the OS max path limit. The fully qualified file name must be less than {1} characters.</value>
<comment></comment>
</data>
<data name="JsonParsingError" UESanitized="false" Visibility="Public">
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Give these a bit of a "namespace" like maybe SolutionFilterJsonParseError.

<value>Path: {0} exceeds the OS max path limit. The fully qualified file name must be less than {1} characters.</value>
<comment></comment>
</data>
<data name="JsonParsingError" UESanitized="false" Visibility="Public">
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aside for @rainersigwald afaik UESanitized has not been used since approx 2004 and can be removed everywhere unless you know differently

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know differently. @tmeschter should we remove 'em all?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can remove them all. They don't affect the .xlf files in any way, and so aren't part of the localization pipeline.

</data>
<data name="MissingSolutionError" UESanitized="false" Visibility="Public">
<value>MSB5026: The solution filter file at "{0}" specifies there will be a solution file at "{1}", but that file does not exist.</value>
<comment>{StrBegin="MSB: 5026"}UE: The solution filename is provided separately to loggers.</comment>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some of these StrBegin are wrong. {StrBegin="MSB: 5026"} should be {StrBegin="MSB5026: "}

@rainersigwald The purpose of these annotations was to support pseudo localizer tools without breaking the processing of the string by removing the necessary error/warning prefix. I do not know whether pseudo localization is still relevant these days -- if not, they can all be removed -- unlreated to this change of course

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The loc team does use annotations like these for purposes beyond pseudo loc; I recommend correcting them when you see issues rather than removing them.

{
ProjectFileErrorUtilities.VerifyThrowInvalidProjectFile
(
false /* just throw the exception */,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is an overload of ProjectFileErrorUtilities.ThrowInvalidProjectFile with 5 parameters that is missing here. It is identical to what is written here but assumes false for the first parameter

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see three ProjectFileErrorUtilities.ThrowInvalidProjectFiles, and they all take bool condition as the first argument.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think Dan's suggesting adding a new overload, and I agree.

Copy link
Contributor Author

@Forgind Forgind Jan 7, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I see. That makes more sense. I was confused because the new method would have four arguments with the fourth having a variable number of parameters.

int visualStudioVersion = 0;
if (FileUtilities.IsSolutionFilterFilename(projectFile))
{
solutionVersion = 16;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's no guarantee that we will rediscover it in time to prevent the future problems, and no way to ensure the TODO is done at a specific time, so it's usually a good idea to address this sort of thing as soon as it's discovered.

However, is this even relevant? Is the change down in line 2074 all we need here, and we just need to get to it? If that's the case can we make the logic say that more clearly?

<comment>{StrBegin="MSB4188: "} Error when the build stops suddenly for some reason. For example, because a child node died.</comment>
</data>
<data name="BuildEngineCallbacksInTaskHostUnsupported" UESanitized="false" Visibility="Public">
<data name="BuildEngineCallbacksInTaskHostUnsupported" Visibility="Public">
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please don't remove UESanitized in this PR; it's not related to this work.

Copy link
Contributor Author

@Forgind Forgind Jan 7, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, but it doesn't hurt, and you previously recommended removing them all. I can do it in another PR, but this felt simpler and more connected to where it was recommended to remove them.

TransientTestFolder folder = testEnvironment.CreateFolder(createFolder: true);
TransientTestFile sln = testEnvironment.CreateFile(folder, "Dev.sln");
TransientTestFile slnf = testEnvironment.CreateFile(folder, "Dev.sln", slnfValue.Replace(@"C:\\notAPath\\MSBuild.Dev.sln", sln.Path.Replace("\\", "\\\\")));
Shouldly.Should.Throw<InvalidProjectFileException>(() => { SolutionFile sf = SolutionFile.Parse(slnf.Path); });
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a using for Shouldly and simplify this reference.

TransientTestFolder folder = testEnvironment.CreateFolder(createFolder: true);
TransientTestFile sln = testEnvironment.CreateFile(folder, "Dev.sln");
TransientTestFile slnf = testEnvironment.CreateFile(folder, "Dev.sln", slnfValue.Replace(@"C:\\notAPath\\MSBuild.Dev.sln", sln.Path.Replace("\\", "\\\\")));
Shouldly.Should.Throw<InvalidProjectFileException>(() => { SolutionFile sf = SolutionFile.Parse(slnf.Path); });
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Shouldly.Should.Throw<InvalidProjectFileException>(() => { SolutionFile sf = SolutionFile.Parse(slnf.Path); });
Shouldly.Should.Throw<InvalidProjectFileException>(() => SolutionFile.Parse(slnf.Path));

")]
public void InvalidSolutionFilters(string slnfValue)
{
Assert.False(File.Exists("C:\\notAPath2\\MSBuild.Dev.sln"));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this relevant? Don't you just replace it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't replace it, which means this test will fail if someone actually makes C:\notAPath2\MSBuild.Dev.sln, and I imagine they would be extremely confused when they went to investigate. This means it will still fail, but the user would have a hint in the right direction. I can replace it with a comment if you'd prefer.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I got confused by notAPath (replaced) vs notAPath2 (tests pointing to a nonexistent solution by absolute path).

This is moot if the solution filters contain relative paths, though, which still seems to me more like what VS does.

using JsonDocument text = JsonDocument.Parse(File.ReadAllText(value));
JsonElement solution = text.RootElement.GetProperty("solution");
_solutionFile = solution.GetProperty("path").GetString();
if (!File.Exists(_solutionFile))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have an abstraction for filesystem that we hope to build some caching into that should be used here:

Suggested change
if (!File.Exists(_solutionFile))
if (!FileSystems.Default.FileExists(_solutionFile))

_solutionFilter = new HashSet<string>(NativeMethodsShared.IsLinux ? StringComparer.Ordinal : StringComparer.OrdinalIgnoreCase);
foreach (JsonElement project in solution.GetProperty("projects").EnumerateArray())
{
if (!File.Exists(Path.Combine(Path.GetDirectoryName(_solutionFile), project.GetString())) && !File.Exists(project.GetString()))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not defer the check to the normal time that the solution would deal with the file?

I would expect an error when the filter's project path isn't a project in the .sln file (that's SolutionFilterFilterContainsProjectNotInSolution, right?), but not when the project doesn't exist on disk (at this point).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because this is a problem with the .slnf, not with the .sln. You're right that that other error is for if there's something in the .slnf that isn't in the .sln.

I think it's valuable to have both types of exceptions. If we let projects exist in the .slnf that don't exist elsewhere, the user could try to modify the .slnf, make a small typo, and be very confused as to why their project wasn't being built.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand what you're saying here. Can you clarify?

The slnf filters the solution, right? So the entries in the slnf should match the projects in the solution. I don't see what presence/absence on disk has to do with anything.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The motivating scenario here is if the user wants to include a project in what is being built, so they add it to the .slnf but make a typo. This throws an error if they do that.

Another way of looking at it is that if a file doesn't exist, we can't build it anyway, so it shouldn't be in the .slnf.


if (_solutionFilter != null)
{
HashSet<string> projectPaths = new HashSet<string>(_projectsInOrder.Count, NativeMethodsShared.IsLinux ? StringComparer.Ordinal : StringComparer.OrdinalIgnoreCase);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't like sprinkling this case-sensitivity check here, since it's wrong in some cases (you can have a case-sensitive FS on macOS, and you can have case-insensitive ones on Linux). I don't think we need to fix that right now (it's basically impossible to get right), but I'd like to at least have it in one place--should we create a NativeMethodsShared.IsFileSystemCaseSensitive or NativeMethodsShared.FileSystemComparer or put a comparer into IFileSystem and use it here? @cdmihai feel free to chime in :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I replaced it with a call to NativeMethodsShared.OSUsesCaseSensitivePaths instead—for now, that just returns NativeMethodsShared.IsLinux, but if we decide it's time to implement a more complicated but more correct solution, this will be silently fixed.

if (_solutionFilter != null)
{
HashSet<string> projectPaths = new HashSet<string>(_projectsInOrder.Count, NativeMethodsShared.IsLinux ? StringComparer.Ordinal : StringComparer.OrdinalIgnoreCase);
for (int a = 0; a < _projectsInOrder.Count; a++)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not a foreach?

<value>MSB5026: The solution filter file at "{0}" specifies there will be a solution file at "{1}", but that file does not exist.</value>
<comment>{StrBegin="MSB5026: "}UE: The solution filename is provided separately to loggers.</comment>
</data>
<data name="SolutionFilterFilterContainsProjectNotInSolution" Visibility="Public">
Copy link

@Mohit-Chakraborty Mohit-Chakraborty Jan 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this result in duplicated errors for a missing project file?
I see that when building a solution we get

MSB3202: The project file "<>" was not found.

for a project in the solution that does not exist.

I guess it's not a duplicated message technically, but that if MSB3202 is still logged, it will have sufficient info.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MSB3202 is for if a project in the .sln doesn't exist on disk; this new error is for if a project is in the filter but is not in the solution file. Together, they ensure that any project in the .slnf is in both the .sln and on disk as desired, but they can never both be thrown because one requires the project to be in the .sln and the other cannot have the project in the .sln.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, got it. Thanks for the explanation.

@Forgind Forgind merged commit ab9b2f3 into dotnet:master Jan 9, 2020
@Forgind Forgind deleted the slnf-filter branch January 9, 2020 23:55
Forgind added a commit to Forgind/msbuild that referenced this pull request Jan 10, 2020
This reverts commit ab9b2f3, reversing
changes made to 86d9494.
Forgind added a commit that referenced this pull request Jan 10, 2020
Revert "Apply a solution filter (#4899)"
@Forgind Forgind mentioned this pull request Jan 10, 2020
Forgind added a commit to Forgind/msbuild that referenced this pull request Feb 10, 2020
This reverts commit ab9b2f3, reversing
changes made to 86d9494.
Forgind added a commit that referenced this pull request Feb 10, 2020
This reverts commit ab9b2f3, reversing
changes made to 86d9494.
Forgind added a commit to Forgind/msbuild that referenced this pull request Feb 10, 2020
This reverts commit ab9b2f3, reversing
changes made to 86d9494.
Forgind added a commit to Forgind/msbuild that referenced this pull request Feb 12, 2020
This reverts commit ab9b2f3, reversing
changes made to 86d9494.
Forgind added a commit that referenced this pull request Feb 12, 2020
* handle duplicate property name

* use string resource, assign next error code

* assert exception message

* use Shouldly as suggested in code review

* split huge line of xml in test

* correct use of Should.Throw

* eliminate _switchesAdded variable

- no significant effect on performance because item count is always low

* Revert "Apply a solution filter (#4899)"

This reverts commit ab9b2f3, reversing
changes made to 86d9494.

Co-authored-by: szaliszali <szaliszali@users.noreply.github.com>
Forgind added a commit that referenced this pull request Feb 24, 2020
This reverts commit ab9b2f3, reversing
changes made to 86d9494.
Forgind added a commit to Forgind/msbuild that referenced this pull request Mar 18, 2020
Forgind added a commit that referenced this pull request Mar 18, 2020
Forgind added a commit that referenced this pull request Jun 3, 2020
* Revert "Revert "Apply a solution filter (#4899)" (#5142)"

This reverts commit 2e4c186.

* Update System.Text.Json to 4.7.0

* Do not ngen System.Text.Json
Forgind added a commit that referenced this pull request Jun 3, 2020
* Revert "Revert "Apply a solution filter (#4899)" (#5142)"

This reverts commit 2e4c186.

* Update System.Text.Json to 4.7.0

* Do not ngen System.Text.Json

* Innocuous change
Forgind added a commit that referenced this pull request Jun 4, 2020
* Revert "Revert "Apply a solution filter (#4899)" (#5142)"

This reverts commit 2e4c186.

* Update System.Text.Json to 4.7.0

* Add pkgdefs
Forgind added a commit that referenced this pull request Jun 5, 2020
* Revert "Revert "Apply a solution filter (#4899)" (#5142)"

This reverts commit 2e4c186.

* Update System.Text.Json to 4.7.0

* Add pkgdefs

* Move to Build
Forgind added a commit to Forgind/msbuild that referenced this pull request Jun 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Consider supporting .slnf solution filters

6 participants