Skip to content

Control embedding Restore files in binlog#5494

Merged
jeffkl merged 4 commits intoNuGet:devfrom
KirillOsenkov:kirillo/noembed
Dec 13, 2023
Merged

Control embedding Restore files in binlog#5494
jeffkl merged 4 commits intoNuGet:devfrom
KirillOsenkov:kirillo/noembed

Conversation

@KirillOsenkov
Copy link
Copy Markdown
Contributor

@KirillOsenkov KirillOsenkov commented Nov 6, 2023

Bug

Fixes: NuGet/Home#12957

Regression? Last working version:

Description

This changes allows users to specify what level of files to embed in the MSBuild binary log.

Value Files to embed
0 or false Do not embed any files
2 Embed everything (dgspec, assets file, g.props, and g.targets)
Any other value Embed assets file, g.props, and g.targets

Users can specify what get embedded via the MSBuild property RestoreEmbedFilesInBinlog:

For example, to disable the functionality:

/p:RestoreEmbedFilesInBinlog=false

Or to opt into embedding everything:

/p:RestoreEmbedFilesInBinlog=2

Any other value embeds just the default (assets file, g.props, and g.targets)

PR Checklist

  • PR has a meaningful title

  • PR has a linked issue.

  • Described changes

  • Tests

    • Automated tests added
    • OR
    • Test exception
    • OR
    • N/A
  • Documentation

    • Documentation PR or issue filled
    • OR
    • N/A

@KirillOsenkov KirillOsenkov requested a review from a team as a code owner November 6, 2023 04:04
@ghost ghost added the Community PRs created by someone not in the NuGet team label Nov 6, 2023
@jeffkl jeffkl self-assigned this Nov 7, 2023
@KirillOsenkov
Copy link
Copy Markdown
Contributor Author

I have tested this by building this PR version of NuGet and inserting into a boostrap version of MSBuild that I also built myself.

I've verified that the two new properties work as expected when passing via command-line: /p:EmbedDGSpecInBinlog=false.

However I've found that setting EmbedDGSpecInBinlog to false in Directory.Build.props doesn't work when restoring a solution, because it doesn't get imported during Restore. Here are the files imported during Restore of the .sln.metaproj:
image

I did find that it works as expected when I create a Directory.Solution.props at the root of the repo:

<Project>
    <PropertyGroup>
        <EmbedDGSpecInBinlog>false</EmbedDGSpecInBinlog>
    </PropertyGroup>
</Project>

It then gets imported during Restore:
image

It also does work as expected when restoring a specific .csproj, Directory.Build.props does get imported in that case.

Another thing I've learned is that even if you set EmbedRestoreFilesInBinlog to false, .nuget.g.props and .nuget.g.targets files will be embedded anyway by virtue of all projects being imported automatically being embedded. We also still embed project.assets.json files by virtue of this logic:

https://github.com/dotnet/sdk/pull/16840/files

I've verified that EmbedProjectAssetsFile still works fine to control that embedding.

So technically to turn off embedding project.assets.json you need to set both EmbedRestoreFilesInBinlog as well as EmbedProjectAssetsFile, because they are now embedded from two separate places. I think this is fine for now, and we can later perhaps just delete the logic from dotnet/sdk#16840.

However this PR works fine for the main goal: to be able to turn off .dgspec files specifically and keep other files, which is the behavior I want for those large binlogs.

@KirillOsenkov
Copy link
Copy Markdown
Contributor Author

Another minor unexpected thing is that setting properties in Directory.Solution.props will only affect the solution restore. Setting EmbedProjectAssetsFile in Directory.Solution.props will have no effect as it won't be imported by individual project builds.

nkolev92
nkolev92 previously approved these changes Nov 9, 2023
@ghost ghost added the Status:No recent activity PRs that have not had any recent activity and will be closed if the label is not removed label Nov 16, 2023
@ghost
Copy link
Copy Markdown

ghost commented Nov 16, 2023

This PR has been automatically marked as stale because it has no activity for 7 days. It will be closed if no further activity occurs within another 90 days of this comment. If it is closed, you may reopen it anytime when you're ready again, as long as you don't delete the branch.

KirillOsenkov added a commit to KirillOsenkov/MSBuildStructuredLog that referenced this pull request Nov 27, 2023
If you open %localappdata%\microsoft\MSBuildStructuredLog\Settings.txt and add IgnoreEmbeddedFiles=.dgspec.json, they won't be read from the binlog, thus saving memory and startup time.

This is primarily to work around NuGet/NuGet.Client#5494, which embeds 600 MB of .nuget.dgspec.json files in a binlog, growing its size from 300 MB -> 900 MB, and slowing down the binlog loading time from 1:50 to 2:30.
@KirillOsenkov
Copy link
Copy Markdown
Contributor Author

@jeffkl this continues to be super painful for us, my proposal is to merge something in to get us unblocked for the next release, and if you like, we can still find the proper nice solution later. I'm worried we'll miss the next train and I'll have to live with 900 MB binlogs for another several months.

@ghost ghost removed the Status:No recent activity PRs that have not had any recent activity and will be closed if the label is not removed label Dec 6, 2023
@jeffkl jeffkl requested a review from nkolev92 December 8, 2023 00:13
@jeffkl
Copy link
Copy Markdown
Contributor

jeffkl commented Dec 8, 2023

@NuGet/nuget-client I have pushed a commit to update this pull request based on a conversation I had with the author. Please review.

donnie-msft
donnie-msft previously approved these changes Dec 8, 2023
Copy link
Copy Markdown
Contributor

@donnie-msft donnie-msft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jeffkl
Copy link
Copy Markdown
Contributor

jeffkl commented Dec 8, 2023

e of the following?

I think this feature is only useful for us when a customer is reporting a problem and we need the dgspec. We would have them restore again with /p:RestoreEmbedFilesInBinlog=2 and then the binlog would contain more files. The dgspec isn't really useful for users and only for us. The assets file is already being embedded by the .NET SDK and the .g.props and g.targets are also being embedded automatically if the exist since they are imported. I'm going to send a follow-up PR to MSBuild to turn that off for restore because the files that get embedded are what are already on disk and not what NuGet generates.

But my opinion is that we don't need to document it at this time since its really only on when you enable the binary logger and these files have already been embedded by default for 2+ years.

Copy link
Copy Markdown
Contributor Author

@KirillOsenkov KirillOsenkov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Adds two new properties on the Restore and RestoreEx tasks:
EmbedRestoreFilesInBinlog and EmbedDGSpecInBinlog

The first one controls whether to embed any files from the Restore task into the binlog at all, the second allows to turn off .dgspec.json files specifically, as these can get quite large. In one case the binlog size increased from 300 MB to 900 MB by including the .dgspec.json files.

Both are true by default to preserve existing behavior.

Fixes NuGet/Home#12957
@jeffkl jeffkl merged commit 2ba7481 into NuGet:dev Dec 13, 2023
@KirillOsenkov KirillOsenkov deleted the kirillo/noembed branch December 14, 2023 01:34
@KirillOsenkov
Copy link
Copy Markdown
Contributor Author

OMG this makes me so happy! I can’t wait to consume this! Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Community PRs created by someone not in the NuGet team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RestoreTask: Control whether to embed files in the binlog

4 participants