Skip to content

Migrate publishing feeds to azure#45416

Merged
dibarbet merged 17 commits intodotnet:masterfrom
dibarbet:migrate_feeds
Jun 30, 2020
Merged

Migrate publishing feeds to azure#45416
dibarbet merged 17 commits intodotnet:masterfrom
dibarbet:migrate_feeds

Conversation

@dibarbet
Copy link
Member

Resolves - #39279

TODO -
Change to master publish data
Get API key for azure feeds and modify pipeline yaml

@dibarbet dibarbet requested a review from a team as a code owner June 24, 2020 03:31
@dibarbet dibarbet requested a review from tmat June 24, 2020 03:31
@dibarbet dibarbet marked this pull request as draft June 24, 2020 03:54

$VSSetupDir = Join-Path $ArtifactsDir "VSSetup\$configuration"
$PackagesDir = Join-Path $ArtifactsDir "packages\$configuration"
$PublishDataUrl = "https://raw.githubusercontent.com/dotnet/roslyn/master/eng/config/PublishData.json"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to change this to master before I merge.

"vssdk": "https://pkgs.dev.azure.com/azure-public/vside/_packaging/vssdk/nuget/v3/index.json",
"vs-impl": "https://pkgs.dev.azure.com/azure-public/vside/_packaging/vs-impl/nuget/v3/index.json"
},
"packages": {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Listing this as a global entry means that it can never differ between our branches. Hence at any point in the future if we decide to change where these packages are published we'll be stuck because all branches will be reading this master list.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see - which would be better then, specifying the per branch mapping as a global entry (as we do currently for the other publishing data) or just reading this map from a place local to the currently running build?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The mapping just needs to be flexible here. Essentially what happens in the future when we need to change master but not break our publishing for release/dev16.8.

Think there are a couple ways to do this:

  1. Just add this data into our existing branch specific entries below. That will work just fine but it does mean that our file is going to start getting big really fast.
  2. Could do some type of diffing system where this is the default and we put deviations in the branch specific entries. Thta may be overkill though.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ended up with something in between 1 and 2:

  1. Added keys in the packages map that can define different sets of package mappings.
  2. Updated the branch configuration to reference the correct key from the packages object.

This way it's possible to change the mappings based on branch and re-use mappings when they don't change. Drawback is that the entire mapping has to be specified again, rather than just the diff. Seems simple enough for now and if we end up needing to modify it a lot we can re-think this.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is a good approach.

@dibarbet
Copy link
Member Author

@jaredpar one additional question - @tmat mentioned that publishing to the dotnet5 feed might be already done by arcade and so doesn't need to be done here. Is that correct?

"vssdk": "https://pkgs.dev.azure.com/azure-public/vside/_packaging/vssdk/nuget/v3/index.json",
"vs-impl": "https://pkgs.dev.azure.com/azure-public/vside/_packaging/vs-impl/nuget/v3/index.json"
},
"packages": {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The mapping just needs to be flexible here. Essentially what happens in the future when we need to change master but not break our publishing for release/dev16.8.

Think there are a couple ways to do this:

  1. Just add this data into our existing branch specific entries below. That will work just fine but it does mean that our file is going to start getting big really fast.
  2. Could do some type of diffing system where this is the default and we put deviations in the branch specific entries. Thta may be overkill though.

@jaredpar
Copy link
Member

@tmat mentioned that publishing to the dotnet5 feed might be already done by arcade and so doesn't need to be done here. Is that correct?

I do not know. In general though I think for the dotnet5 feed publish we should use the arcade infrastruture if possibel

@dibarbet
Copy link
Member Author

dibarbet commented Jun 24, 2020

Think this is mostly ready -
I've verified that the command output looks generally correct (output below). I can't actually verify until this is merged and a signed build is run with this since

  1. we can only upload real signed packages to those feeds and
  2. the authentication is a pipeline step.
Downloading https://raw.githubusercontent.com/dibarbet/roslyn/b83a09493dbf0b2f8b8f086eb1408707208982c6/eng/config/PublishData.json
  Publishing Microsoft.CodeAnalysis.3.7.0-dev.nupkg
    Skipping publishing for Microsoft.CodeAnalysis.3.7.0-dev.nupkg as it is published by arcade
  Publishing Microsoft.CodeAnalysis.Build.Tasks.3.7.0-dev.nupkg
    Skipping publishing for Microsoft.CodeAnalysis.Build.Tasks.3.7.0-dev.nupkg as it is published by arcade
  Publishing Microsoft.CodeAnalysis.Common.3.7.0-dev.nupkg
    Skipping publishing for Microsoft.CodeAnalysis.Common.3.7.0-dev.nupkg as it is published by arcade
  Publishing Microsoft.CodeAnalysis.Compilers.3.7.0-dev.nupkg
    Skipping publishing for Microsoft.CodeAnalysis.Compilers.3.7.0-dev.nupkg as it is published by arcade
  Publishing Microsoft.CodeAnalysis.CSharp.3.7.0-dev.nupkg
    Skipping publishing for Microsoft.CodeAnalysis.CSharp.3.7.0-dev.nupkg as it is published by arcade
  Publishing Microsoft.CodeAnalysis.CSharp.CodeStyle.3.7.0-dev.nupkg
    Skipping publishing for Microsoft.CodeAnalysis.CSharp.CodeStyle.3.7.0-dev.nupkg as it is published by arcade
  Publishing Microsoft.CodeAnalysis.CSharp.EditorFeatures.3.7.0-dev.nupkg
    Test command: nuget push Microsoft.CodeAnalysis.CSharp.EditorFeatures.3.7.0-dev.nupkg --source https://pkgs.dev.azure.com/azure-public/vside/_packaging/vssdk/nuget/v3/index.json --api-key AzureArtifacts
  Publishing Microsoft.CodeAnalysis.CSharp.Features.3.7.0-dev.nupkg
    Skipping publishing for Microsoft.CodeAnalysis.CSharp.Features.3.7.0-dev.nupkg as it is published by arcade
  Publishing Microsoft.CodeAnalysis.CSharp.Scripting.3.7.0-dev.nupkg
    Skipping publishing for Microsoft.CodeAnalysis.CSharp.Scripting.3.7.0-dev.nupkg as it is published by arcade
  Publishing Microsoft.CodeAnalysis.CSharp.Workspaces.3.7.0-dev.nupkg
    Skipping publishing for Microsoft.CodeAnalysis.CSharp.Workspaces.3.7.0-dev.nupkg as it is published by arcade
  Publishing Microsoft.CodeAnalysis.Debugging.3.7.0-dev.nupkg
    Skipping publishing for Microsoft.CodeAnalysis.Debugging.3.7.0-dev.nupkg as it is published by arcade
  Publishing Microsoft.CodeAnalysis.EditorFeatures.3.7.0-dev.nupkg
    Test command: nuget push Microsoft.CodeAnalysis.EditorFeatures.3.7.0-dev.nupkg --source https://pkgs.dev.azure.com/azure-public/vside/_packaging/vssdk/nuget/v3/index.json --api-key AzureArtifacts
  Publishing Microsoft.CodeAnalysis.EditorFeatures.Common.3.7.0-dev.nupkg
    Test command: nuget push Microsoft.CodeAnalysis.EditorFeatures.Common.3.7.0-dev.nupkg --source https://pkgs.dev.azure.com/azure-public/vside/_packaging/vssdk/nuget/v3/index.json --api-key AzureArtifacts
  Publishing Microsoft.CodeAnalysis.EditorFeatures.Text.3.7.0-dev.nupkg
    Test command: nuget push Microsoft.CodeAnalysis.EditorFeatures.Text.3.7.0-dev.nupkg --source https://pkgs.dev.azure.com/azure-public/vside/_packaging/vssdk/nuget/v3/index.json --api-key AzureArtifacts
  Publishing Microsoft.CodeAnalysis.EditorFeatures.Wpf.3.7.0-dev.nupkg
    Test command: nuget push Microsoft.CodeAnalysis.EditorFeatures.Wpf.3.7.0-dev.nupkg --source https://pkgs.dev.azure.com/azure-public/vside/_packaging/vssdk/nuget/v3/index.json --api-key AzureArtifacts
  Publishing Microsoft.CodeAnalysis.ExternalAccess.Apex.3.7.0-dev.nupkg
    Test command: nuget push Microsoft.CodeAnalysis.ExternalAccess.Apex.3.7.0-dev.nupkg --source https://pkgs.dev.azure.com/azure-public/vside/_packaging/vs-impl/nuget/v3/index.json --api-key AzureArtifacts
  Publishing Microsoft.CodeAnalysis.ExternalAccess.Debugger.3.7.0-dev.nupkg
    Test command: nuget push Microsoft.CodeAnalysis.ExternalAccess.Debugger.3.7.0-dev.nupkg --source https://pkgs.dev.azure.com/azure-public/vside/_packaging/vs-impl/nuget/v3/index.json --api-key AzureArtifacts
  Publishing Microsoft.CodeAnalysis.ExternalAccess.FSharp.3.7.0-dev.nupkg
    Test command: nuget push Microsoft.CodeAnalysis.ExternalAccess.FSharp.3.7.0-dev.nupkg --source https://pkgs.dev.azure.com/azure-public/vside/_packaging/vs-impl/nuget/v3/index.json --api-key AzureArtifacts
  Publishing Microsoft.CodeAnalysis.ExternalAccess.Razor.3.7.0-dev.nupkg
    Test command: nuget push Microsoft.CodeAnalysis.ExternalAccess.Razor.3.7.0-dev.nupkg --source https://pkgs.dev.azure.com/azure-public/vside/_packaging/vs-impl/nuget/v3/index.json --api-key AzureArtifacts
  Publishing Microsoft.CodeAnalysis.ExternalAccess.Xamarin.Remote.3.7.0-dev.nupkg
    Test command: nuget push Microsoft.CodeAnalysis.ExternalAccess.Xamarin.Remote.3.7.0-dev.nupkg --source https://pkgs.dev.azure.com/azure-public/vside/_packaging/vs-impl/nuget/v3/index.json --api-key AzureArtifacts
  Publishing Microsoft.CodeAnalysis.Features.3.7.0-dev.nupkg
    Skipping publishing for Microsoft.CodeAnalysis.Features.3.7.0-dev.nupkg as it is published by arcade
  Publishing Microsoft.CodeAnalysis.InteractiveHost.3.7.0-dev.nupkg
    Test command: nuget push Microsoft.CodeAnalysis.InteractiveHost.3.7.0-dev.nupkg --source https://pkgs.dev.azure.com/azure-public/vside/_packaging/vssdk/nuget/v3/index.json --api-key AzureArtifacts
  Publishing Microsoft.CodeAnalysis.LanguageServer.Protocol.3.7.0-dev.nupkg
    Test command: nuget push Microsoft.CodeAnalysis.LanguageServer.Protocol.3.7.0-dev.nupkg --source https://pkgs.dev.azure.com/azure-public/vside/_packaging/vs-impl/nuget/v3/index.json --api-key AzureArtifacts
  Publishing Microsoft.CodeAnalysis.PooledObjects.3.7.0-dev.nupkg
    Skipping publishing for Microsoft.CodeAnalysis.PooledObjects.3.7.0-dev.nupkg as it is published by arcade
  Publishing Microsoft.CodeAnalysis.Remote.Razor.ServiceHub.3.7.0-dev.nupkg
    Test command: nuget push Microsoft.CodeAnalysis.Remote.Razor.ServiceHub.3.7.0-dev.nupkg --source https://pkgs.dev.azure.com/azure-public/vside/_packaging/vs-impl/nuget/v3/index.json --api-key AzureArtifacts
  Publishing Microsoft.CodeAnalysis.Remote.ServiceHub.3.7.0-dev.nupkg
    Test command: nuget push Microsoft.CodeAnalysis.Remote.ServiceHub.3.7.0-dev.nupkg --source https://pkgs.dev.azure.com/azure-public/vside/_packaging/vs-impl/nuget/v3/index.json --api-key AzureArtifacts
  Publishing Microsoft.CodeAnalysis.Remote.Workspaces.3.7.0-dev.nupkg
    Test command: nuget push Microsoft.CodeAnalysis.Remote.Workspaces.3.7.0-dev.nupkg --source https://pkgs.dev.azure.com/azure-public/vside/_packaging/vs-impl/nuget/v3/index.json --api-key AzureArtifacts
  Publishing Microsoft.CodeAnalysis.Scripting.3.7.0-dev.nupkg
    Skipping publishing for Microsoft.CodeAnalysis.Scripting.3.7.0-dev.nupkg as it is published by arcade
  Publishing Microsoft.CodeAnalysis.Scripting.Common.3.7.0-dev.nupkg
    Skipping publishing for Microsoft.CodeAnalysis.Scripting.Common.3.7.0-dev.nupkg as it is published by arcade
  Publishing Microsoft.CodeAnalysis.VisualBasic.3.7.0-dev.nupkg
    Skipping publishing for Microsoft.CodeAnalysis.VisualBasic.3.7.0-dev.nupkg as it is published by arcade
  Publishing Microsoft.CodeAnalysis.VisualBasic.CodeStyle.3.7.0-dev.nupkg
    Skipping publishing for Microsoft.CodeAnalysis.VisualBasic.CodeStyle.3.7.0-dev.nupkg as it is published by arcade
  Publishing Microsoft.CodeAnalysis.VisualBasic.EditorFeatures.3.7.0-dev.nupkg
    Test command: nuget push Microsoft.CodeAnalysis.VisualBasic.EditorFeatures.3.7.0-dev.nupkg --source https://pkgs.dev.azure.com/azure-public/vside/_packaging/vssdk/nuget/v3/index.json --api-key AzureArtifacts
  Publishing Microsoft.CodeAnalysis.VisualBasic.Features.3.7.0-dev.nupkg
    Skipping publishing for Microsoft.CodeAnalysis.VisualBasic.Features.3.7.0-dev.nupkg as it is published by arcade
  Publishing Microsoft.CodeAnalysis.VisualBasic.Workspaces.3.7.0-dev.nupkg
    Skipping publishing for Microsoft.CodeAnalysis.VisualBasic.Workspaces.3.7.0-dev.nupkg as it is published by arcade
  Publishing Microsoft.CodeAnalysis.Workspaces.Common.3.7.0-dev.nupkg
    Skipping publishing for Microsoft.CodeAnalysis.Workspaces.Common.3.7.0-dev.nupkg as it is published by arcade
  Publishing Microsoft.CodeAnalysis.Workspaces.MSBuild.3.7.0-dev.nupkg
    Skipping publishing for Microsoft.CodeAnalysis.Workspaces.MSBuild.3.7.0-dev.nupkg as it is published by arcade
  Publishing Microsoft.Net.Compilers.3.7.0-dev.nupkg
    Skipping publishing for Microsoft.Net.Compilers.3.7.0-dev.nupkg as it is published by arcade
  Publishing Microsoft.Net.Compilers.Toolset.3.7.0-dev.nupkg
    Skipping publishing for Microsoft.Net.Compilers.Toolset.3.7.0-dev.nupkg as it is published by arcade
  Publishing Microsoft.NETCore.Compilers.3.7.0-dev.nupkg
    Skipping publishing for Microsoft.NETCore.Compilers.3.7.0-dev.nupkg as it is published by arcade
  Publishing Microsoft.VisualStudio.LanguageServices.3.7.0-dev.nupkg
    Test command: nuget push Microsoft.VisualStudio.LanguageServices.3.7.0-dev.nupkg --source https://pkgs.dev.azure.com/azure-public/vside/_packaging/vssdk/nuget/v3/index.json --api-key AzureArtifacts
  Publishing Microsoft.VisualStudio.LanguageServices.LiveShare.3.7.0-dev.nupkg
    Test command: nuget push Microsoft.VisualStudio.LanguageServices.LiveShare.3.7.0-dev.nupkg --source https://pkgs.dev.azure.com/azure-public/vside/_packaging/vs-impl/nuget/v3/index.json --api-key AzureArtifacts
  Publishing Microsoft.VisualStudio.LanguageServices.Razor.RemoteClient.3.7.0-dev.nupkg
    Test command: nuget push Microsoft.VisualStudio.LanguageServices.Razor.RemoteClient.3.7.0-dev.nupkg --source https://pkgs.dev.azure.com/azure-public/vside/_packaging/vs-impl/nuget/v3/index.json --api-key AzureArtifacts
  Publishing Microsoft.CodeAnalysis.Lsif.Generator.3.7.0-dev.nupkg
    Test command: nuget push Microsoft.CodeAnalysis.Lsif.Generator.3.7.0-dev.nupkg --source https://pkgs.dev.azure.com/azure-public/vside/_packaging/vs-impl/nuget/v3/index.json --api-key AzureArtifacts
  Publishing Microsoft.VisualStudio.IntegrationTest.Utilities.3.7.0-dev.nupkg
    Test command: nuget push Microsoft.VisualStudio.IntegrationTest.Utilities.3.7.0-dev.nupkg --source https://pkgs.dev.azure.com/azure-public/vside/_packaging/vs-impl/nuget/v3/index.json --api-key AzureArtifacts

@dibarbet dibarbet marked this pull request as ready for review June 24, 2020 23:15
},
"packages": {
"default": {
"Microsoft.CodeAnalysis" : "arcade",
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted the script to throw if it found any packages that did not have a properly configured feed, so I kept these mapped to arcade to show that these are intentionally not being uploaded.

@dibarbet dibarbet requested a review from a team June 25, 2020 20:47
@dibarbet
Copy link
Member Author

I'm going to hold off on merging this until after the snap Monday.

@dibarbet dibarbet merged commit df147ca into dotnet:master Jun 30, 2020
@ghost ghost added this to the Next milestone Jun 30, 2020
@dibarbet dibarbet deleted the migrate_feeds branch June 30, 2020 18:20
@JoeRobich JoeRobich modified the milestones: Next, 16.8.P1 Jul 20, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants