Generate sbom for arcade by epananth · Pull Request #8424 · dotnet/arcade

epananth · 2022-02-02T22:07:58Z

To double check:

The right tests are in and and the right validation has happened. Guidance: https://github.com/dotnet/core-eng/tree/master/Documentation/Validation

Generate sbom for arcade and repos using arcade

Repos using job.yml will not have to make any change on their end. They will just need an arcade update to generate sbom
Repos who do not use job.yml will have to import steps/template/generate-sbom.yml in their pipeline yml

epananth · 2022-02-02T23:39:29Z

Arcade test -> https://dev.azure.com/dnceng/internal/_build/results?buildId=1588768&view=logs&s=6884a131-87da-5381-61f3-d7acc3b91d76

epananth · 2022-02-03T02:26:35Z

Removed flag enableSbom as we need this in all repos

Sdk test -> https://dev.azure.com/dnceng/internal/_build/results?buildId=1588939&view=logs&j=fa59fe4e-195c-56fa-189b-58fd241f10dd

eng/common/templates/steps/generate-sbom.yml

eng/common/templates/job/job.yml

eng/common/templates/steps/generate-sbom.yml

epananth · 2022-02-03T22:43:04Z

arcade test-> https://dev.azure.com/dnceng/internal/_build/results?buildId=1590521&view=results

sdk test-> https://dev.azure.com/dnceng/internal/_build/results?buildId=1590720&view=results

eng/common/templates/job/job.yml

eng/common/templates/steps/generate-sbom.yml

eng/common/templates/job/job.yml

eng/common/templates/steps/generate-sbom.yml

riarenas · 2022-02-04T15:56:49Z

Could you test this in a repo that has multiple build legs? Both arcade and SDK only have a windows leg.

epananth · 2022-02-10T06:30:37Z

arcade test -> https://dev.azure.com/dnceng/internal/_build/results?buildId=1604133&view=results

epananth · 2022-02-10T06:31:13Z

aspnetcore test -> https://dev.azure.com/dnceng/internal/_build/results?buildId=1604274&view=results

eng/common/templates/job/job.yml

riarenas

Some comments.

Your aspnetcore test failed to generate all linux sboms due to execute permissions for the prep script. Does it still have the execute flag set in git?

EDIT: Also, I'd like us to keep trying to use a common variables template so that we don't have to copy the same defaults in 3 different templates, but if it's proving difficult it's fine as a followup PR.

riarenas · 2022-02-10T15:40:55Z

eng/common/templates/job/job.yml

      continueOnError: ${{ parameters.continueOnError }}
      condition: and(succeeded(), eq(variables['_DotNetPublishToBlobFeed'], 'true'))
+
+  - ${{ if and(eq(parameters.runAsPublic, 'false'), ne(variables['System.TeamProject'], 'public'), notin(variables['Build.Reason'], 'PullRequest')) }}:


Wondering why you're using two conditions instead of adding the sbom parameter check to this one?

we do the same for microbuild, where we have enableMicrobuild in job.yml, so I used the same format... I will move it other condition

riarenas · 2022-02-10T15:41:48Z

eng/common/templates/job/job.yml

+    - ${{ if eq(parameters.enableSbom, 'true') }}:
+      - template: /eng/common/templates/steps/generate-sbom.yml
+        parameters:
+          PackageVersion : $ {{ parameters.packageVersion}}


Suggested change

PackageVersion : $ {{ parameters.packageVersion}}

PackageVersion : ${{ parameters.packageVersion}}

Updated this

riarenas · 2022-02-10T15:42:24Z

eng/common/templates/jobs/jobs.yml

  enableSourceIndex: false
  sourceIndexParams: {}
-
+  


[nit] trailing space it looks like?

riarenas · 2022-02-10T15:42:56Z

eng/common/templates/job/job.yml

  name: ''
  preSteps: []
  runAsPublic: false
+#Sbom related params


Suggested change

#Sbom related params

# Sbom related params

Here and everywhere

added a space inbetween the comment and #

riarenas · 2022-02-10T15:43:30Z

eng/common/templates/jobs/jobs.yml

-            name: NetCore1ESPool-Internal
-            demands: ImageOverride -equals Build.Server.Amd64.VS2019
-
+          vmImage: 'windows-2019'


Make sure to not check in this testing change.

removed this unwanted code

riarenas · 2022-02-10T15:44:33Z

eng/common/templates/steps/generate-sbom.yml

+# ManifestDirPath - The path of the directory where the generated manifest files will be placed
+
+parameters:
+  PackageVersion: ''


The default is missing for the version.

And for the BuildDropPath.

Added above params in generete-sbom.yml

riarenas · 2022-02-10T17:32:26Z

eng/common/templates/steps/generate-sbom.yml

+  sbomContinueOnError: true
+
+steps:
+- powershell: |


These scripts are still being called as if they were inline. You can take a look at how checked in scripts should be used with the tasks in a bunch of our other templates, such as:

arcade/eng/common/templates/job/publish-build-assets.yml

Lines 58 to 62 in c2af37f

- task: PowerShell@2

displayName: Enable cross-org NuGet feed authentication

inputs:

filePath: $(Build.SourcesDirectory)/eng/common/enable-cross-org-publishing.ps1

arguments: -token $(dn-bot-all-orgs-artifact-feeds-rw)

Updated to use the task instead of inline script

epananth · 2022-02-10T18:47:41Z

aspnetcore test -> https://dev.azure.com/dnceng/internal/_build/results?buildId=1604274&view=results

Some comments.

Your aspnetcore test failed to generate all linux sboms due to execute permissions for the prep script. Does it still have the execute flag set in git?

EDIT: Also, I'd like us to keep trying to use a common variables template so that we don't have to copy the same defaults in 3 different templates, but if it's proving difficult it's fine as a followup PR.

I added the link, it was late at night and didn't stick around for the entire test. I did add the execute script before I ran it last night. That did not work, so I will have to add chmod back in the script.

I agree I need more time to test that template, I don't want to rush it.

riarenas · 2022-02-10T18:50:03Z

That did not work, so I will have to add chmod back in the script.

You mentioned you had a test where it had worked?

riarenas · 2022-02-10T18:52:13Z

I wonder if the Linux failure is related to my other comment of calling the task incorrectly?

epananth · 2022-02-10T18:53:27Z

I did a test in arcade, unfortunately that does not have a linux leg. So it not work.

riarenas · 2022-02-10T18:56:26Z

I just checked out this branch: https://dev.azure.com/dnceng/internal/_git/dotnet-aspnetcore?version=GBsbom

 git ls-files -s eng/common/generate-sbom-prep.sh
100644 d7d132c8d8fd8dfc081c9ea144baf11a621ac0c7 0       eng/common/generate-sbom-prep.sh

So the file is not marked as executable. Make sure to run the command I shared before and commit the file again.

epananth · 2022-02-10T23:16:25Z

Arcade test -> https://dev.azure.com/dnceng/internal/_build/results?buildId=1605919&view=logs&j=fa59fe4e-195c-56fa-189b-58fd241f10dd&t=fa59fe4e-195c-56fa-189b-58fd241f10dd

Aspnet core test -> https://dev.azure.com/dnceng/internal/_build/results?buildId=1605974&view=logs&j=4a1f0fe1-d87a-57fd-c29d-71ea15e38f30&t=dd5521f1-f2eb-55e5-cb58-87c5968347ac

riarenas

Assuming all tests work I think all I have remaining are nits.
Let's create a followup issue to make the templates source their default values from a common location though.

eng/common/templates/job/job.yml

riarenas · 2022-02-10T23:11:12Z

eng/common/templates/steps/generate-sbom.yml

+  condition: or(eq(variables['Agent.Os'], 'Windows_NT'), eq(variables['Agent.Os'], 'Darwin'))
+  inputs: 
+    filePath: ./eng/common/generate-sbom-prep.ps1
+    arguments: ${{parameters.manifestDirPath}}


Don't you need to pass -ManifestDirPath first so the script knows which parameter it is? or does this work because there's a single argument to the script?

Co-authored-by: Ricardo Arenas <riarenas@microsoft.com>

epananth · 2022-02-10T23:21:13Z

this happened :(

I def want that approval pls !!!!

ghost · 2022-02-10T23:23:19Z

Hello @epananth!

Because this pull request has the auto-merge label, I will be glad to assist with helping to merge this pull request once all check-in policies pass.

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (`@msftbot`) and give me an instruction to get started! Learn more here.

mmitche · 2022-02-11T15:13:14Z

WOOHOO!

ericstj · 2022-02-12T02:11:56Z

Nice! Trying this out in machinelearning now dotnet/machinelearning#6076.

epananth added 7 commits February 1, 2022 18:08

Generate sbom

3570931

typo

c659632

change artifact name

2a785ee

Use step template

8dc16cf

update

43509da

minor fix

04f56bc

revert enableSbom flag

9d04302

epananth marked this pull request as ready for review February 2, 2022 23:38

Test

a2552ca

epananth requested review from mmitche and riarenas February 3, 2022 04:46

mmitche reviewed Feb 3, 2022

View reviewed changes

eng/common/templates/steps/generate-sbom.yml Outdated Show resolved Hide resolved

mmitche reviewed Feb 3, 2022

View reviewed changes

eng/common/templates/job/job.yml Outdated Show resolved Hide resolved

mmitche reviewed Feb 3, 2022

View reviewed changes

eng/common/templates/steps/generate-sbom.yml Outdated Show resolved Hide resolved

epananth added 3 commits February 3, 2022 10:19

Review comments

0ef8bae

include packages

2ab9d44

update path

726951c

mmitche reviewed Feb 3, 2022

View reviewed changes

eng/common/templates/job/job.yml Outdated Show resolved Hide resolved

mmitche reviewed Feb 3, 2022

View reviewed changes

eng/common/templates/steps/generate-sbom.yml Outdated Show resolved Hide resolved

epananth added 4 commits February 3, 2022 15:57

comments

c456f3e

remove BuildComponent

aef1b28

revert

d5e8351

Merge branch 'main' of https://github.com/dotnet/arcade into sbom-task

94312fc

mmitche previously approved these changes Feb 4, 2022

View reviewed changes

riarenas reviewed Feb 4, 2022

View reviewed changes

eng/common/templates/job/job.yml Show resolved Hide resolved

riarenas reviewed Feb 4, 2022

View reviewed changes

eng/common/templates/steps/generate-sbom.yml Outdated Show resolved Hide resolved

epananth added 2 commits February 9, 2022 19:34

readding _

cf5a796

Keeping it consistent with bash script

fa524c8

riarenas reviewed Feb 10, 2022

View reviewed changes

eng/common/templates/job/job.yml Show resolved Hide resolved

riarenas reviewed Feb 10, 2022

View reviewed changes

epananth added 6 commits February 10, 2022 12:35

More review comments

414377b

Merge branch 'main' of https://github.com/dotnet/arcade into sbom-task

bd2fa83

Update jobs template

394e8a7

add space to comments

841c0d8

Update condition for powershell task

5339428

Add an extra space

0b13ab6

riarenas previously approved these changes Feb 10, 2022

View reviewed changes

epananth dismissed riarenas’s stale review via 002a0d0 February 10, 2022 23:19

Update eng/common/templates/job/job.yml

002a0d0

Co-authored-by: Ricardo Arenas <riarenas@microsoft.com>

epananth requested a review from riarenas February 10, 2022 23:22

riarenas approved these changes Feb 10, 2022

View reviewed changes

epananth added the auto-merge Automatically merge PR once CI passes. label Feb 10, 2022

ghost merged commit 78eaf78 into dotnet:main Feb 11, 2022

This pull request was closed.

	PackageVersion : $ {{ parameters.packageVersion}}
	PackageVersion : ${{ parameters.packageVersion}}

	- task: PowerShell@2
	displayName: Enable cross-org NuGet feed authentication
	inputs:
	filePath: $(Build.SourcesDirectory)/eng/common/enable-cross-org-publishing.ps1
	arguments: -token $(dn-bot-all-orgs-artifact-feeds-rw)

Conversation

epananth commented Feb 2, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

To double check:

Uh oh!

epananth commented Feb 2, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

epananth commented Feb 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

epananth commented Feb 3, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

riarenas commented Feb 4, 2022

Uh oh!

epananth commented Feb 10, 2022

Uh oh!

epananth commented Feb 10, 2022

Uh oh!

Uh oh!

riarenas left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

epananth commented Feb 10, 2022

Uh oh!

riarenas commented Feb 10, 2022

Uh oh!

riarenas commented Feb 10, 2022

Uh oh!

epananth commented Feb 10, 2022

Uh oh!

riarenas commented Feb 10, 2022

Uh oh!

epananth commented Feb 10, 2022

Uh oh!

riarenas left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

epananth commented Feb 10, 2022

Uh oh!

ghost commented Feb 10, 2022

epananth commented Feb 2, 2022 •

edited

Loading

epananth commented Feb 2, 2022 •

edited

Loading

epananth commented Feb 3, 2022 •

edited

Loading

riarenas left a comment •

edited

Loading

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (`@msftbot`) and give me an instruction to get started! Learn more here.