Integrate security release approval into release pipeline by willdollman · Pull Request #63990 · sourcegraph/sourcegraph-public-snapshot

willdollman · 2024-07-22T15:47:53Z

As part of the Vuln Scanning Improvements project, I've been working on tooling to automate the security approval step of the release process.

This PR integrates these improvements into the release pipeline:

Internal releases will run a vulnerability scan
Promote-to-public releases will check for security approval

If a public release does not have security approval, it will block the promotion process. The step happens at the start of the pipeline so should be a fast-fail. You can also check for release approval before running promotion by running @secbot cve approve-release <version> in the #secbot-commands channel. In an ideal world we (security) will have already gone through and approved ahead of release.

I've tested this PR as much as I can without running an actual release! We have a 5.5.x release tomorrow so it'll be a good test. If it does cause problems that can't be easily solved, it can always be temporarily disabled.

I've tagged this PR to be backported to 5.5.x.

Pre-merge checklist

Revert commit that disables release promotion

Test plan

Manual testing of the release process:

Successful test run that shows the security scan being triggered
Promote to public test run that shows the security approval approving a release
Promote to public test run that shows the security approval rejecting a release with un-accepted CVEs

Changelog

Chickensoupwithrice · 2024-07-22T18:17:34Z

+      - name: 'Check Security Approval'
+        cmd: |
+          curl --location 'https://security-manager.sgdev.org/approve-release?release={{tag}}' --header "Authorization: Bearer ${SCANNER_TOKEN}" --fail && "Release {{tag}} has security approval"
+            || echo "Release {{tag}} does not have security approval - reach out to the Security Team to resolve."


These steps would run after promotion has happened. We should move them to the cmd that runs the buildkite pipeline

Very true! But that cmd runs locally, so the sec-manager webhook secret won't be available. I've moved them to the runtype.PromoteRelease Buildkite pipeline, though this makes it a little harder to bypass the release approval step in the event something goes wrong.

+// checkSecurityApproval checks whether the specified release has release approval from the Security Team.
+func checkSecurityApproval(c Config) operations.Operation {
+	return func(pipeline *bk.Pipeline) {
+		pipeline.AddStep("Check security approval",
+			bk.Agent("queue", AspectWorkflows.QueueDefault),
+			bk.Env("VERSION", c.Version),
+			bk.AnnotatedCmd(
+				"./tools/release/check_security_approval.sh",
+				bk.AnnotatedCmdOpts{
+					Annotations: &bk.AnnotationOpts{
+						Type:         bk.AnnotationTypeInfo,
+						IncludeNames: false,
+					},
+				},
+			),
+		)


-          curl --location 'https://incoming.sgdev.org/new-image-scan?tag={{tag}}&scanType=release&dev=true' --header 'X-Special-Header: ${SCANNER_TOKEN}'
+          set -eu
+
+          curl --location 'https://security-manager.sgdev.org/internal-release-scan?release={{tag}}' --request POST --header "Authorization: Bearer ${SECURITY_SCANNER_TOKEN}"


+curl --location "https://security-manager.sgdev.org/approve-release?release=${VERSION}" \
+  --header "Authorization: Bearer ${SECURITY_SCANNER_TOKEN}" --fail
+SECURITY_APPROVAL=$?
+
+if [ "$SECURITY_APPROVAL" -eq 0 ]; then
+  echo "Release \`${VERSION}\` has security approval." | tee -a ./annotations/security_approval.md
+else
+  echo -e "Release ${VERSION} does **not** have security approval - reach out to the Security Team to resolve.\n" | tee -a ./annotations/security_approval.md
+  echo "Run \`@SecBot cve approve-release 5.5.1339\` in [#secbot-commands](https://sourcegraph.slack.com/archives/C07BQJDFCV8) to check the approval status." | tee -a ./annotations/security_approval.md
+  exit 1


+curl --location "https://security-manager.sgdev.org/approve-release?release=${VERSION}" \
+  --header "Authorization: Bearer ${SECURITY_SCANNER_TOKEN}" --fail
+SECURITY_APPROVAL=$?
+
+if [ "$SECURITY_APPROVAL" -eq 0 ]; then
+  echo "Release \`${VERSION}\` has security approval." | tee -a ./annotations/security_approval.md
+else
+  echo -e "Release ${VERSION} does **not** have security approval - reach out to the Security Team to resolve.\n" | tee -a ./annotations/security_approval.md
+  echo "Run \`@SecBot cve approve-release 5.5.1339\` in [#secbot-commands](https://sourcegraph.slack.com/archives/C07BQJDFCV8) to check the approval status." | tee -a ./annotations/security_approval.md
+  exit 1
+fi


Chickensoupwithrice

Looks great! 🚀 Excited to take it for a spin tomorrow :)

As part of the [Vuln Scanning Improvements](https://linear.app/sourcegraph/project/[p0]-vulnerability-scanning-improvements-75299c4312dd/issues) project, I've been working on tooling to automate the security approval step of the release process. This PR integrates these improvements into the release pipeline: * Internal releases will run a vulnerability scan * Promote-to-public releases will check for security approval If a public release does not have security approval, it will block the promotion process. The step happens at the start of the pipeline so should be a fast-fail. You can also check for release approval before running promotion by running `@secbot cve approve-release <version>` in the #secbot-commands channel. In an ideal world we (security) will have already gone through and approved ahead of release. I've tested this PR as much as I can without running an actual release! We have a 5.5.x release tomorrow so it'll be a good test. If it does cause problems that can't be easily solved, it can always be temporarily disabled. I've tagged this PR to be backported to `5.5.x`.  ## Pre-merge checklist - [x] Revert commit that disables release promotion ## Test plan Manual testing of the release process: - [x] [Successful test run](https://buildkite.com/sourcegraph/sourcegraph/builds/283774#0190dfd6-fa70-4cea-9711-f5b8493c7714) that shows the security scan being triggered - [x] [Promote to public test run](https://buildkite.com/sourcegraph/sourcegraph/builds/283826) that shows the security approval approving a release - [x] [Promote to public test run](https://buildkite.com/sourcegraph/sourcegraph/builds/283817#0190e0ec-0641-4451-b7c7-171e664a3127) that shows the security approval rejecting a release with un-accepted CVEs  ## Changelog  (cherry picked from commit 9dd901f)

…eline (#64030) As part of the [Vuln Scanning Improvements](https://linear.app/sourcegraph/project/[p0]-vulnerability-scanning-improvements-75299c4312dd/issues) project, I've been working on tooling to automate the security approval step of the release process. This PR integrates these improvements into the release pipeline: * Internal releases will run a vulnerability scan * Promote-to-public releases will check for security approval If a public release does not have security approval, it will block the promotion process. The step happens at the start of the pipeline so should be a fast-fail. You can also check for release approval before running promotion by running `@secbot cve approve-release <version>` in the #secbot-commands channel. In an ideal world we (security) will have already gone through and approved ahead of release. I've tested this PR as much as I can without running an actual release! We have a 5.5.x release tomorrow so it'll be a good test. If it does cause problems that can't be easily solved, it can always be temporarily disabled. I've tagged this PR to be backported to `5.5.x`. ## Pre-merge checklist - [x] Revert commit that disables release promotion ## Test plan Manual testing of the release process: - [x] [Successful test run](https://buildkite.com/sourcegraph/sourcegraph/builds/283774#0190dfd6-fa70-4cea-9711-f5b8493c7714) that shows the security scan being triggered - [x] [Promote to public test run](https://buildkite.com/sourcegraph/sourcegraph/builds/283826) that shows the security approval approving a release - [x] [Promote to public test run](https://buildkite.com/sourcegraph/sourcegraph/builds/283817#0190e0ec-0641-4451-b7c7-171e664a3127) that shows the security approval rejecting a release with un-accepted CVEs ## Changelog <br> Backport 9dd901f from #63990 Co-authored-by: Will Dollman <will.dollman@sourcegraph.com>

Integrate security release approval into release pipeline

e00d2b3

willdollman self-assigned this Jul 22, 2024

cla-bot Bot added the cla-signed label Jul 22, 2024

github-advanced-security AI found potential problems Jul 22, 2024

View reviewed changes

Comment thread release.yaml Fixed

Comment thread release.yaml Fixed

Chickensoupwithrice reviewed Jul 22, 2024

View reviewed changes

willdollman added 4 commits July 23, 2024 09:39

Add rough initial release step to check security approval

5fc6b4f

Remove release approval step in release.yaml

46a1931

Rename secret to match value already configured in buildkite

9af60e5

Add annotations and remove useless sprintf

8a43fd3

github-advanced-security AI found potential problems Jul 23, 2024

View reviewed changes

willdollman added 4 commits July 23, 2024 10:35

Update token name to match buildkite secret

8edbbed

Set eu on internal-release-scan

8f0e295

Merge branch 'main' into will/security-integration-release

798ee42

Disable relese promotion in runtype.PromoteRelease pipeline

95eb89d

github-advanced-security AI found potential problems Jul 23, 2024

View reviewed changes

Tidy up check_security_approval

5b6ec2d

github-advanced-security AI found potential problems Jul 23, 2024

View reviewed changes

Comment thread tools/release/check_security_approval.sh Fixed

willdollman added 6 commits July 23, 2024 17:40

Exit if release does not have security approval

b7edf8c

Comment out promoteToPublic finalization steps

46efa4d

Delete finalization

83b218a

Tweak security check script

f5d0c20

Improve annotation

aef99be

Newlines

fc65506

github-advanced-security AI found potential problems Jul 23, 2024

View reviewed changes

Comment thread tools/release/check_security_approval.sh Fixed

willdollman added 4 commits July 23, 2024 19:39

Tweak curl output

d576073

Fixup annotation + curl

12d54e1

Emoji sprinkles

446f794

Revert changes to prevent public promotion

2a139b9

willdollman force-pushed the will/security-integration-release branch from 37d9b5c to 2a139b9 Compare July 23, 2024 19:08

Merge branch 'main' into will/security-integration-release

b7bc893

willdollman marked this pull request as ready for review July 23, 2024 19:09

github-advanced-security AI found potential problems Jul 23, 2024

View reviewed changes

willdollman added the backport 5.5.x label Jul 23, 2024

willdollman requested a review from Chickensoupwithrice July 23, 2024 19:23

Chickensoupwithrice approved these changes Jul 23, 2024

View reviewed changes

willdollman merged commit 9dd901f into main Jul 24, 2024

willdollman deleted the will/security-integration-release branch July 24, 2024 08:19

sourcegraph-release-bot mentioned this pull request Jul 24, 2024

[Backport 5.5.x] Integrate security release approval into release pipeline #64030

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integrate security release approval into release pipeline#63990

Integrate security release approval into release pipeline#63990
willdollman merged 21 commits into
mainfrom
will/security-integration-release

willdollman commented Jul 22, 2024 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Chickensoupwithrice Jul 22, 2024

Uh oh!

willdollman Jul 23, 2024 •

edited

Loading

Uh oh!

Check notice

Uh oh!

Check notice

Uh oh!

Uh oh!

Check notice

Check notice

Chickensoupwithrice left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

willdollman commented Jul 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pre-merge checklist

Test plan

Changelog

Uh oh!

Uh oh!

Uh oh!

Chickensoupwithrice Jul 22, 2024

Choose a reason for hiding this comment

Uh oh!

willdollman Jul 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Check notice

Uh oh!

Check notice

Uh oh!

Uh oh!

Check notice

Check notice

Chickensoupwithrice left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

willdollman commented Jul 22, 2024 •

edited

Loading

willdollman Jul 23, 2024 •

edited

Loading