Downloading artifacts before publishing is exceeding 10gb, in runtime #6987
Description
- This issue is blocking
- This issue is causing unreasonable pain
Build -> https://dev.azure.com/dnceng/internal/_build/results?buildId=1002147&view=results
Symbols packages are copied to a temporary directory and then uploaded to Symbol server, this was done cos we wanted blobUpload and symbolUpload have to work parallelly and SymUploader need read permission when uploading to symbol server.
But now in Runtime
• We are downloading 6.3 gigs of artifacts to publish: https://dev.azure.com/dnceng/internal/_build/results?buildId=1002323&view=logs&j=ba23343f-f710-5af9-782d-5bd26b102304&t=cecdd0ae-1666-570d-4521-d44005725bca&l=3166
• The symbol packages are 4.36 GB, so when we do that new intermediate copy step, we’re going over the 10 gigs in a hosted machine.
ckages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21117.1\tools\SdkTasks\PublishArtifactsInManifest.proj(105,5): error : There is not enough space on the disk. : 'D:\a\1\a\tempSymbols\Microsoft.NETCore.App.Runtime.Mono.osx-x64.6.0.0-preview.2.21118.9.symbols.nupkg'
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21117.1\tools\SdkTasks\PublishArtifactsInManifest.proj(105,5): error : at System.IO.FileSystem.CopyFile(String sourceFullPath, String destFullPath, Boolean overwrite)
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21117.1\tools\SdkTasks\PublishArtifactsInManifest.proj(105,5): error : at Microsoft.DotNet.Build.Tasks.Feed.PublishArtifactsInManifestV3.CopySymbolFilesToTemporaryLocation(BuildModel buildModel, String symbolTemporaryLocation) in /_/src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifestV3.cs:line 228
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21117.1\tools\SdkTasks\PublishArtifactsInManifest.proj(105,5): error : at Microsoft.DotNet.Build.Tasks.Feed.PublishArtifactsInManifestV3.ExecuteAsync() in /_/src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifestV3.cs:line 95
##[error].packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21117.1\tools\SdkTasks\PublishArtifactsInManifest.proj(105,5): error : (NETCORE_ENGINEERING_TELEMETRY=Publish) There is not enough space on the disk. : 'D:\a\1\a\tempSymbols\Microsoft.NETCore.App.Runtime.Mono.osx-x64.6.0.0-preview.2.21118.9.symbols.nupkg'
at System.IO.FileSystem.CopyFile(String sourceFullPath, String destFullPath, Boolean overwrite)
at Microsoft.DotNet.Build.Tasks.Feed.PublishArtifactsInManifestV3.CopySymbolFilesToTemporaryLocation(BuildModel buildModel, String symbolTemporaryLocation) in /_/src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifestV3.cs:line 228
at Microsoft.DotNet.Build.Tasks.Feed.PublishArtifactsInManifestV3.ExecuteAsync() in /_/src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifestV3.cs:line 95
[Edit : renamed this as its not blocking but we need a better way to handle this scenario ]