Publish artifacts and symbols one file at a time#7086

Merged

epananth merged 39 commits intodotnet:mainfrom

epananth:publishing-pr

Apr 12, 2021

Member

epananth commented Mar 10, 2021 •

edited

Loading

To double check:

The right tests are in and and the right validation has happened. Guidance: https://github.com/dotnet/core-eng/tree/master/Documentation/Validation

Issue -> #6987

Both the old functionality to download the entire artifact folder is still present and can be turned on by enabling the artifact download task and switching UseStreamingPublishing= false in the publish-assets.yml

There is definitely some redundant code, I'm hoping code review will help me make it better.

NOTE : PdbArtifacts are still downloaded like we did earlier, cos there is not way to get the file names before hand to download them from PdbArtifacts directory.

epananth added 2 commits

March 10, 2021 10:05


          Improve publishing performance

2ca72c8


          remove unwanted param

epananth changed the title ~~Publishing pr~~ Publish artifacts and symbols one file at a time

epananth added 9 commits

March 10, 2021 14:17


          some fixes

9d3a66e


          more update

271e5c2


          minor fixes

23fd68d


          Merge branch 'main' of https://github.com/dotnet/arcade into publishi…

086bc12

…ng-pr


          Merge branch 'main' of https://github.com/dotnet/arcade into publishi…

583c310

…ng-pr


          Disabled package download

b82c1c0

fix

ff093e1


          Remove some logging

91f474a


          More fix

68e8ea8

Member Author

epananth commented Mar 16, 2021 •

edited

Loading

Ran some tests

Runtime Latest build -> https://dnceng.visualstudio.com/internal/_build/results?buildId=1040969&view=results
vs Runtime build with new hosted pool -> https://dnceng.visualstudio.com/internal/_build/results?buildId=1040959&view=results

Other promotion builds
dotnet-sdk -
https://dnceng.visualstudio.com/internal/_build/results?buildId=1040973&view=logs&j=ba23343f-f710-5af9-782d-5bd26b102304&t=6e277ba4-1c1e-552d-b96f-db0aeb4be20a

dotnet-winforms -
https://dnceng.visualstudio.com/internal/_build/results?buildId=1040984&view=logs&j=ba23343f-f710-5af9-782d-5bd26b102304&t=6e277ba4-1c1e-552d-b96f-db0aeb4be20a

epananth marked this pull request as ready for review

March 16, 2021 07:51


          Merge branch 'main' of https://github.com/dotnet/arcade into publishi…

7c1325a

…ng-pr

Member Author

epananth commented Mar 16, 2021

More test results from day before

https://dnceng.visualstudio.com/internal/_build/results?buildId=1039004&view=logs&j=ba23343f-f710-5af9-782d-5bd26b102304&t=6e277ba4-1c1e-552d-b96f-db0aeb4be20a - 20 m 48 seconds
https://dnceng.visualstudio.com/internal/_build/results?buildId=1039011&view=logs&j=ba23343f-f710-5af9-782d-5bd26b102304&t=6e277ba4-1c1e-552d-b96f-db0aeb4be20a - 20 m 46 seconds
https://dnceng.visualstudio.com/internal/_build/results?buildId=1039034&view=logs&j=ba23343f-f710-5af9-782d-5bd26b102304&t=6e277ba4-1c1e-552d-b96f-db0aeb4be20a - 21 m 42 seconds

this download with the new hosted pool took 31 minutes https://dnceng.visualstudio.com/internal/_build/results?buildId=1038957&view=logs&j=ba23343f-f710-5af9-782d-5bd26b102304&t=cecdd0ae-1666-570d-4521-d44005725bca, compared to the whole thing taking https://dev.azure.com/dnceng/internal/_build/results?buildId=1039004&view=logs&j=ba23343f-f710-5af9-782d-5bd26b102304&t=6e277ba4-1c1e-552d-b96f-db0aeb4be20a 20 m 48 seconds..

Member Author

epananth commented Mar 16, 2021

Odd one

Runtime took 31 mins
https://dnceng.visualstudio.com/internal/_build/results?buildId=1040978&view=logs&j=ba23343f-f710-5af9-782d-5bd26b102304&t=6e277ba4-1c1e-552d-b96f-db0aeb4be20a

Member Author

epananth commented Mar 16, 2021

I ran tests for different promotions parallelly, and they all succeeded. I have also added retry logic for downloading files. So I think I covered most of the tests, we had discussed.

riarenas reviewed

View reviewed changes

eng/publishing/v3/publish-assets.yml Outdated Show resolved Hide resolved

riarenas reviewed

View reviewed changes

Contributor

riarenas left a comment

First round of feedback. Still need to finish looking at all the files.

src/Microsoft.DotNet.Build.Tasks.Feed/src/BuildArtifacts.cs Show resolved Hide resolved

src/Microsoft.DotNet.Build.Tasks.Feed/src/BuildArtifacts.cs Show resolved Hide resolved

src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifestBase.cs Outdated Show resolved Hide resolved

src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifestBase.cs Outdated Show resolved Hide resolved

src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifestBase.cs Outdated Show resolved Hide resolved

src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifestBase.cs Outdated Show resolved Hide resolved

src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifestBase.cs Show resolved Hide resolved

src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifestBase.cs Outdated Show resolved Hide resolved

src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifestBase.cs Outdated Show resolved Hide resolved

src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifestBase.cs Show resolved Hide resolved

riarenas reviewed

View reviewed changes

src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifestBase.cs Outdated Show resolved Hide resolved

riarenas reviewed

View reviewed changes

src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifestBase.cs Outdated Show resolved Hide resolved

riarenas reviewed

View reviewed changes

Contributor

riarenas left a comment

Second pass. Two files to go

src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifestBase.cs Outdated Show resolved Hide resolved

src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifestBase.cs Outdated Show resolved Hide resolved

src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifestBase.cs Outdated Show resolved Hide resolved

src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifestBase.cs Outdated Show resolved Hide resolved

src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifestBase.cs Outdated Show resolved Hide resolved

src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifestBase.cs Show resolved Hide resolved

src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifestBase.cs Outdated Show resolved Hide resolved

src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifestBase.cs Outdated Show resolved Hide resolved

src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifestBase.cs Outdated Show resolved Hide resolved

src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifestBase.cs Outdated Show resolved Hide resolved

epananth added 7 commits

March 17, 2021 18:48


          Merge branch 'main' of https://github.com/dotnet/arcade into publishi…

30e72f6

…ng-pr


          test sdk

037401b


          test

6d73f0b


          review comments

ff239e8


          merge conflict

42edf24


          test

696ae33


          review comments:

3cc26df

epananth requested review from michellemcdaniel and riarenas and removed request for riarenas

April 5, 2021 18:23

riarenas reviewed

View reviewed changes

src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifestBase.cs Outdated Show resolved Hide resolved

riarenas reviewed

View reviewed changes

src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifestBase.cs Show resolved Hide resolved

riarenas approved these changes

View reviewed changes

Contributor

riarenas left a comment

I only have a couple more nits and comments but provided you have some test builds with these latest changes that we can take a look at, this LGTM,


          review comment

86215a9

mmitche reviewed

View reviewed changes

src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifestBase.cs Outdated

                       /// </summary>
                       /// <param name="client">Azdo client</param>
-                      /// <param name="artifact">If it is PackageArtifacts or BlobArtifacts</param>
+                      /// <param name="ArtifactName">If it is PackageArtifacts or BlobArtifacts</param>

Member

mmitche Apr 6, 2021

Nit: the parameter name is artifactName (casing)

Member Author

epananth Apr 6, 2021

Updated this

mmitche reviewed

View reviewed changes

src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifestBase.cs Outdated


		}

		private async Task PublishPackagesToAzureStorageNugetByDownloadingEntireFolderAsync(HashSet<PackageArtifactModel> packagesToPublish,

Member

mmitche Apr 6, 2021

YAY!

mmitche approved these changes

View reviewed changes

Member

mmitche left a comment

LGTM. Thanks for iterating on this. Complicated stuff.

michellemcdaniel approved these changes

View reviewed changes

Contributor

michellemcdaniel left a comment

I'm happy with how it looks in the staging pipeline (the only failures make sense/are because the files had already been uploaded since we tried on a bar id that already has been published). LGTM

Member Author

epananth commented Apr 6, 2021

Latest test against runtime

Publishing	Link	Time taken to publish
New infra Bar Id - 85015	https://dev.azure.com/dnceng/internal/_build/results?buildId=1073502&view=logs&j=ba23343f-f710-5af9-782d-5bd26b102304&t=6e277ba4-1c1e-552d-b96f-db0aeb4be20a	24mins 56s
old infra Bar Id - 85015	https://dev.azure.com/dnceng/internal/_build/results?buildId=1072291&view=logs&j=ba23343f-f710-5af9-782d-5bd26b102304&t=6e277ba4-1c1e-552d-b96f-db0aeb4be20a	23mins 44secs + 3 mins 48 secs
New infra (bar id - 84977)	https://dev.azure.com/dnceng/internal/_build/results?buildId=1073701&view=logs&j=33961d73-1e6b-5306-1d3b-c6c083bff7de&t=577fb5b9-bfb7-5430-eb7a-5c5120b89b4d	25 mins 51 sec
old infra (bar id - 84977)	https://dev.azure.com/dnceng/internal/_build/results?buildId=1071605&view=logs&j=ba23343f-f710-5af9-782d-5bd26b102304&t=6e277ba4-1c1e-552d-b96f-db0aeb4be20a	31 mins 32 sec + 3 min 8 sec


          last change

8de357d

This was referenced Apr 6, 2021

Downloading artifacts before publishing is exceeding 10gb, in runtime #6987

Closed

Unblock runtime publishing #6988

Closed

Member Author

epananth commented Apr 7, 2021

I did some test against installer and I am currently facing some IO exception, used by another process for blob publishing. I am investigating this issue, so I am not going to merge this PR till that issue is fixed.

epananth added 2 commits

April 8, 2021 14:13


          IO exception fix

8a5d431


          test fix

07f63c4

Member Author

epananth commented Apr 8, 2021

Aspnet core -> https://dev.azure.com/dnceng/internal/_build/results?buildId=1079097&view=results
Installer -> https://dev.azure.com/dnceng/internal/_build/results?buildId=1079093&view=logs&j=ba23343f-f710-5af9-782d-5bd26b102304&t=6e277ba4-1c1e-552d-b96f-db0aeb4be20a
Runtime -> https://dev.azure.com/dnceng/internal/_build/results?buildId=1079083&view=logs&s=380ab8c6-4139-55ff-15c4-fb48260f4dca&j=ba23343f-f710-5af9-782d-5bd26b102304

Member Author

epananth commented Apr 8, 2021 •

edited

Loading

In the last change, I added the following

Added await to CompareLocalPackageToFeedPackage
Making sure PackageArchiveReader was using the correct scope
added retry to ContainerIdAsync
More logs for downloading file
Added retry to directory delete
Updated the test

epananth requested review from michellemcdaniel and riarenas

April 8, 2021 23:39

Member Author

epananth commented Apr 9, 2021

@mmitche mentioned my change is good to merge. :)

Member Author

epananth commented Apr 9, 2021

Installer build first time - https://dev.azure.com/dnceng/internal/_build/results?buildId=1079093&view=logs&j=ba23343f-f710-5af9-782d-5bd26b102304&t=6e277ba4-1c1e-552d-b96f-db0aeb4be20a

did delete-build-from-channel and retried build - https://dev.azure.com/dnceng/internal/_build/results?buildId=1078127&view=results

riarenas approved these changes

View reviewed changes

epananth merged commit 23b4c45 into dotnet:main

akoeplinger pushed a commit to akoeplinger/arcade that referenced this pull request


          Publish artifacts and symbols one file at a time (dotnet#7086)

0a92032

* Improve publishing performance

epananth deleted the publishing-pr branch

April 15, 2021 05:30

danmoseley mentioned this pull request

Add RuntimeConfiguration.NotRelease danmoseley/arcade#1

Closed

1 task

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet