Use GitHub Actions for release automation by bk2204 · Pull Request #3840 · git-lfs/git-lfs

bk2204 · 2019-09-26T21:29:17Z

Currently, our release process is highly manual and doing a release can consume a significant amount of a maintainer's day. Automate the process of doing a release by building and uploading assets on GitHub Actions.

There's a lot of code here; the series is best explained by reading the commit messages, which cover the changes better than this summary ever could.

PastelMobileSuit

This looks fantastic, thank you for automating this process! This should make things so much easier for everyone ✨

I had a couple bash-related questions but once those are answered I'll approve this

PastelMobileSuit · 2019-09-30T22:41:16Z

Makefile


+# TRIMPATH contains arguments to be passed to go to strip paths on Go 1.13 and
+# newer.
+TRIMPATH ?= $(shell [ "$$(go version | awk '{print $$3}' | sed -e 's/^.*\.//')" -ge 13 ] && echo -trimpath)


Could you explain this line and what the bash here does? I'm not a bash expert, so I'm having some trouble parsing it to give a review!

Sure. It gets the Go version and if it's 1.13 or above (we grab the 1.13 text out of go version using awk (which prints the third whitespace-separated field) and strip out everything including the dot and before, then compare to 13), we set the variable to -trimpath. That's because that option is only available in Go 1.13 or later.

If it's not (we're on Go 1.12 or earlier), nothing happens, and the variable is empty.

Note that in this command, the $ characters are doubled due to the makefile, so the command we're actually executing is the following:

[ "$(go version | awk '{print $3}' | sed -e 's/^.*\.//')" -ge 13 ] && echo -trimpath

I just realized there's a minor bug here; I neglected to consider the patch version (since Homebrew's version didn't have that), so I'll push up a fix with that in a second. The internal pipeline will look like this:

go version | awk '{print $3}' | sed -e 's/^[^.]*\.//;s/\..*$//;'

which basically does the same thing, just picking out the part in the middle of go1.13.1 (the 13) or the 13 in go1.13.

Thanks for the explanation! I was thrown off by the $$, I didn't realize this was a requirement in Makefiles.

I've tested with both go1.13 and go1.13.1 and that new sed invocation seems to be working!

Yeah, since Makefiles have variables introduced by $ as well, you have to double it if you want to write a literal $ so it can distinguish between its variables and the shell's..

PastelMobileSuit · 2019-09-30T23:39:38Z

script/upload

+  local url=$(curl https://api.github.com/repos/$REPO/releases | \
+    jq -r '.[] | select(.name == "'"$version"'") | .url')
+
+  [ -n "$url" ]


What happens here if the test returns false?

Since we're running with set -e, the script aborts if any command (other than the condition of a conditional) exits unsuccessfully. So in this case, the script would exit if the test were false.

This is more of a sanity check (equivalent to an assert) than anything else: we want to be sure that there is some release that we're operating on, or we're going to have a bad time.

I'll add an abort here so that there's a useful error message.

It's important for us to flag this binary as a test tool so that Go doesn't complain about us having multiple instances of the main function in different files. Not doing this causes failures when the Debian package is built in the Docker containers, since the Debian build system wants to build all packages, including the test packages, at once.

When building packages in containers, we pass options to docker. However, when running in a noninteractive environment such as GitHub Actions, we can't use the -i option, since standard input will not be a TTY and docker will consequently fail. Check for a TTY and pass the -i option only if one is present.

Not all users want to write a token to disk. In addition, specifying the token like this allows us to use this script on GitHub Actions.

When we automate the release process using GitHub Actions, we'll want to upload an unsigned list of hashes that the maintainer can then sign and re-upload. In order to do so, we'll need to handle the case where we're uploading an unsigned list of hashes, so teach our upload script about plain files named "sha256sums".

While the ability to verify assets is important to ensuring our releases are intact, we also want to be able to skip this step if we're running in GitHub Actions. We may upload assets in multiple steps and not want to verify immediately, or even have a signed manifest until the final, manual stages of the process. Add a --skip-verify flag that allows skipping the verification process.

In the future, we'll want to use this script solely to download assets and stage them for future processes. In preparation for that, split the download function into a separate location.

We already have the changelog in CHANGELOG.md, so there's no point in requiring the user to provide it. Let's extract it automatically from the existing file.

When the build and upload process is automated, the only thing left for the maintainer to do will be to download the assets, sign the hashes, and upload the signed manifest. To make this easy and mostly automated, add an option to the upload script, --finalize, that downloads assets, optionally allows the maintainer to inspect them, and then uploads the signed manifest to the GitHub release.

One persistent problem we've had with building binaries is that paths from the build system get embedded in them. We'd like to produce reproducible binaries, so take advantage of Go 1.13's -trimpath argument that does this for us. This is better than the old technique because it works regardless of where our binaries are built on the system and regardless of where GOPATH is located.

Currently, we build Windows objects on a dedicated VM which already has the certificate imported. When we build on GitHub Actions, we'll need to build binaries using a certificate and password specified on the command line. To do so, add several environment variables that allow specifying these values, and if they're specified, pass appropriate arguments on the command line. Note that we hide the output from these commands so as not to leak secrets into the log. Additionally, add a target to the Makefile which can be used to write a Base64-encoded certificate to a file on the file system without echoing it so we don't leak even the encrypted certificate to the log.

When we sign Windows binaries currently, we assume that the user has loaded the certificate into their system credential store. In the past, when binaries were built on bespoke VMs, that was a legitimate design, but when building binaries in an automated fashion, this is no longer suitable. Switch the command line arguments to use dashes instead of slashes, since Git Bash treats options with leading slashes as paths and rewrites them, breaking our invocation. Ensure we don't echo the password in the output; while GitHub actions should filter things for us, it's better not to rely on that. Add the -debug option to help us troubleshoot problems if they occur.

When we create a tarball with Windows assets, we create it with the extension ".tar.gz" and we later on extract it with the -z flag to tar. However, we fail to actually compress the tarball itself. This causes no problems with macOS's tar, which automatically handles compression based on magic numbers, but it breaks with GNU tar, which we use during the Windows build process. Ensure that we actually compress the tarball so that we can decompress it in a later stage.

Currently, our releases are a highly intricate, mostly manual process. Some work on automating the process has already occurred, but there is still a lot of work to be done manually. This means that a release can eat up several hours of a maintainer's day, which in turn prevents us from being able to do patch releases as frequently as we'd like. Automate almost the entire build process by running it on GitHub Actions. This process builds the Windows assets, builds the Linux packages, and prepares and uploads the release assets to GitHub. All that remains is for the maintainer to generate and sign the manifest themselves and upload it, a process which can be easily handled by running script/upload with a few arguments.

Now that we have a much more automated process for building releases, let's update the documentation to tell people what they still have to do and remove mentions of things that have been automated.

PastelMobileSuit

✨

Since PR git-lfs#3840 introduced a GitHub Actions workflow to automate much of the process of releasing a new version of Git LFS, that workflow has concluded by running the script/upload script, which makes requests to GitHub's API to create a new draft release announcement and attach the binary release assets built by the workflow to the announcement. This step necessarily requires that the GITHUB_TOKEN credential provided to the workflow have permission to create a release, which implies fairly broad write access to the repository contents as GitHub does not offer a more fine-grained permission model for this particular task. In order to ensure that our GitHub Actions workflows can not be misused, we would much prefer to run them without write permissions to our repositories. We therefore revise the "build-main" job in our release workflow to remove the step which tries to run the script/upload script, and replace it with steps that instead create a "release-assets" artifact containing the binary release assets built by the workflow. The administrator performing the release may then download this artifact and unpack it, after which they can run the script/upload script manually using their own personal GitHub credentials. We update our release process documentation to explain this new approach and also clarify a few other details of the release process. This change does make our release process slightly more manual, and if in the future GitHub provides a convenient way to enable a single job to have just sufficient permissions to create a draft release announcement, we can revisit this decision in the future.

In PR git-lfs#3840 we introduced a GitHub Actions workflow to build our release assets when we push a new version tag. As a final step in this workflow we run our script/packagecloud.rb script, which requires the use of the "packagecloud-ruby" Ruby gem. At the time, these requirements meant we needed to first execute the actions/setup-ruby action so we could then install the gem and run our Ruby script. Later, in PR git-lfs#4230, we added steps to our CI workflow to build our manual pages, and since these required the use of the "ronn" Ruby gem, we also added the actions/setup-ruby action to our CI workflow. (We subsequently migrated our manual page source files to the AsciiDoc format, in PR git-lfs#5054, but we continue to use Ruby and the "asciidoctor" gem to build our manual pages.) Then in commit b7fa3a5 of PR git-lfs#5236 we upgraded both of our Actions workflows to use the ruby/setup-ruby action instead of the now-deprecated actions/setup-ruby one. Because the ruby/setup-ruby action installs the MSYS2 environment on Windows and sets several key environment variables like PATH and TMPDIR, we also introduced steps to make sure our CI and release processes continued to work as expected in this context, by clearing the TMPDIR variable and renaming the directory containing the MSYS2 executables. Fortunately, the default runners provided by GitHub Actions for the macOS, Windows, and Ubuntu Linux platforms are now all provisioned with an installation of Ruby 3.x which includes the "gem" utility we need to install the "asciidoctor" and "packagecloud-ruby" gems. As these are all we require to run our Ruby scripts, we no longer need the more extensive Ruby development environment provided by the ruby/setup-ruby action. We can therefore simply remove the ruby/setup-ruby steps from our workflows, along with all the special handling of the MSYS2 environment and the TMPDIR environment variable on Windows.

bk2204 requested a review from a team September 26, 2019 21:29

bk2204 force-pushed the actions-release branch from b9c47fc to c49eedc Compare September 26, 2019 21:37

PastelMobileSuit reviewed Oct 1, 2019

View reviewed changes

bk2204 added 15 commits October 1, 2019 14:29

script/upload: allow specifying token via GITHUB_TOKEN

80eb30f

Not all users want to write a token to disk. In addition, specifying the token like this allows us to use this script on GitHub Actions.

script/upload: split downloading assets into a separate function

2f4a861

In the future, we'll want to use this script solely to download assets and stage them for future processes. In preparation for that, split the download function into a separate location.

script/upload: extract changelog automatically

99c206b

We already have the changelog in CHANGELOG.md, so there's no point in requiring the user to provide it. Let's extract it automatically from the existing file.

script/upload: move sanity checks into a separate function

ef92706

docs: update release howto with new process

c8814c7

Now that we have a much more automated process for building releases, let's update the documentation to tell people what they still have to do and remove mentions of things that have been automated.

bk2204 force-pushed the actions-release branch from c49eedc to c8814c7 Compare October 1, 2019 14:29

PastelMobileSuit approved these changes Oct 1, 2019

View reviewed changes

bk2204 merged commit 8d20f78 into git-lfs:master Oct 1, 2019

bk2204 deleted the actions-release branch October 1, 2019 19:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use GitHub Actions for release automation#3840

Use GitHub Actions for release automation#3840
bk2204 merged 15 commits intogit-lfs:masterfrom
bk2204:actions-release

bk2204 commented Sep 26, 2019

Uh oh!

PastelMobileSuit left a comment

Uh oh!

PastelMobileSuit Sep 30, 2019

Uh oh!

bk2204 Oct 1, 2019

Uh oh!

bk2204 Oct 1, 2019

Uh oh!

PastelMobileSuit Oct 1, 2019

Uh oh!

bk2204 Oct 1, 2019

Uh oh!

PastelMobileSuit Sep 30, 2019

Uh oh!

bk2204 Oct 1, 2019

Uh oh!

bk2204 Oct 1, 2019

Uh oh!

PastelMobileSuit left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

bk2204 commented Sep 26, 2019

Uh oh!

PastelMobileSuit left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PastelMobileSuit left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants