Skip to content

Adjust the tag-git workflow and create a release-git workflow to build Git for Windows fully using GitHub Actions#38

Merged
dscho merged 52 commits intomainfrom
release
Mar 7, 2023
Merged

Adjust the tag-git workflow and create a release-git workflow to build Git for Windows fully using GitHub Actions#38
dscho merged 52 commits intomainfrom
release

Conversation

@dscho
Copy link
Member

@dscho dscho commented Feb 28, 2023

  • adjust the tag-git workflow so that it can tag full Git for Windows versions, not just snapshots
  • adjust the git-artifacts workflow to mirror Check Runs to git-for-windows/git (needs to encode the architecture as part of the Check Run name to avoid them overwriting each other)
  • reimplement the Azure Release Pipeline as new GitHub workflow, called release-git

This addresses #37

@dscho dscho self-assigned this Feb 28, 2023
@dscho dscho linked an issue Feb 28, 2023 that may be closed by this pull request
@dscho dscho force-pushed the release branch 2 times, most recently from 178c716 to 8dfbfb8 Compare March 1, 2023 00:12
@dscho dscho force-pushed the release branch 5 times, most recently from 51c242a to b11caa3 Compare March 1, 2023 23:12
One of the main reasons of the existence of the
`git-for-windows-automation` repository is the functionality where the
`build-and-deploy` workflow "mirrors" a Check Run into a repository that
is different from the one that actually runs the workflow.

This allows us to version the workflow definition implementing automated
builds independently of the code that is actually being built.

We will also use this functionality in the `git-artifacts` workflow (and
related workflows).

So let's make the code as reusable as possible by creating a new GitHub
Action specifically for the purpose.

To make things a bit more convenient, the Action maintains a state in a
file in `runner.temp`. This file contains, among other things, the
access token as well as the information when it expires. Since that is
sensitive information and should not be accessible by the build
processes that are run as part of the rest of the workflow, the state
file is encrypted (using the GitHub App's private key that we need to
generate an access token, anyway).

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
@dscho dscho force-pushed the release branch 19 times, most recently from f4b45a7 to dfa7845 Compare March 4, 2023 09:18
dscho added 24 commits March 7, 2023 20:21
This is needed e.g. to download workflow jobs' logs or artifacts.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This makes it easier to organize and encapsulate the functionality.

This commit is best viewed with `--color-moved`.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This should make the code both easier to read as well as to modify.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Git for Windows publishes two NuGet packages (x86_64 only) as part of
its release process. Which means that the `git-artifacts` workflow must
generate them.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
…cessary

As part of the regular `makepkg-mingw` run, we already have a bare clone
in `/usr/src/MINGW-packages/mingw-w64-git/git` in addition to the
checkout in `/usr/src/MINGW-packages/mingw-w64-git/src/git`.

No need to have yet another clone. We can operate on the checkout
directly.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
These checksums are needed to complete the text of the announcement mail
as well as the text of the GitHub Release.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
We need this because we want to know what is the target commit as early
as possible so that we can mirror a Check Run to that target commit.

This commit is best viewed with `--color-moved`.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The `actions/checkout` Action is a more canonical way to do things, and
it also benefits from better performance (shallow by default, clones
only the main branch, etc).

While at it, check out the repository in the top-level workspace (we are
no longer using that location for the `git-for-windows/git` checkout).

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Just like we do in the `build-and-deploy` and the `tag-git` workflows,
we now also mirror a Check Run to `git-for-windows/git` when running the
`git-artifacts` workflow.

Since we want to have one workflow run per architecture, we include the
architecture in the Check Run name, so that all of them show up in the
corresponding git-for-windows/git PR in the Checks tab.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The entire idea of caching the package, once built, is to avoid having
to rebuild the Pacman package just because some _other_ step in the
workflow failed.

But using `actions/cache` as-is won't actually cache the artifact
properly (or worse, it can cache it incompletely when the _build_
failed).

Let's be more careful by using the combination of `actions/cache/save`
and `actions/cache/restore`.

While at it, skip other steps, too, when they are not needed because a
cached package was restored.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Naturally, when the version is re-tagged, we have to rebuild the
package... But previously, we did not, because the `tag-git` workflow
run ID did not play any role in the decision whether to restore or
rebuild. Now it does.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
…t` ran

The `tag-git` workflow, on which the `git-artifacts` runs for the
various architectures rely, publishes a Git bundle to update the release
notes. To make use of that, the `git-artifacts` workflow needs to check
out the `build-extra` repository and merge those changes.

However, it _does_ happen from time to time that the main branch of
`build-extra` advances between the `tag-git` and the `git-artifacts`
run.

Therefore, we need to tell Git to make a merge commit if necessary.

Which means that we have to have Git committer information ready to go,
and therefore we configure `user.*` in the `artifacts` job when building
an `installer`, too.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The `-busybox` infix is not automatically inserted.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
If building from a tag that was generated in a given `tag-git` workflow
run, we must first make sure that that workflow run finished and
succeeded.

This is particularly important because the plan is to teach the
GitForWindowsHelper GitHub App to no longer trigger the `Git Artifacts`
Azure Pipeline upon `/git-artifacts`, but instead to trigger a `tag-git`
workflow run, wait for it to finish, and then to trigger the
`git-artifacts` workflow runs (one for each architecture supported by
Git for Windows).

And the logic to wait for the workflow run to finish and then to trigger
the `git-artifacts` workflow runs is a bit tricky: the actual `tag-git`
workflow run does not provide the necessary metadata (the
git-for-windows/git commit to which to mirror the Check Run) that would
be accessible by the regular `get-a-workflow-run` GitHub REST API call.

Therefore, the plan is not actually to wait for the `tag-git` _workflow
run_ in `git-for-windows-automation` to complete, but instead for the
`tag-git` _Check Run_ that is mirrored into `git-for-windows/git`. This
Check Run is created and updated explicitly and therefore we could make
sure that the relevant information is included there (the `output.text`
attribute contains a link to the workflow run, including its ID, and the
`head_sha` attribute points to the git-for-windows/git commit to which
the newly generated tag points).

Since that Check Run is marked as completed by the workflow run that is
obviously still running, the webhook event might arrive, and the
`git-artifacts` workflow runs might be triggered, _before_ the `tag-git`
workflow run actually finished. Hence the need to potentially await its
completion.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
If we ever need to mirror Check Runs into _multiple_ repositories, this
change will allow that: by specifying a state file for a different repo,
it is possible to maintain multiple states at the same time.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
When an entire file needs to be uploaded, it is probably not a good idea
to read the contents into memory and pass them as parameter. It is much
better to open a stream and pipe it through.

To support that, we allow to pass a file read stream instead of a string
or an object.

Sadly, we cannot use arbitrary read stream, it has to be a file read
stream, because we need to know the file name and the file length.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The Git for Windows project previously used an Azure Release Pipeline to
publish new versions.

This was very convenient, as Azure Release Pipelines not only allow
versioning the Pipeline definition independently of the code that is
being deployed, it allows the individual _stages_ to be versioned
independently and even _while deploying_.

This comes in handy when changes outside of Git for Windows' control
require changes in the stage definition. For example, when the NuGet
deployment process changed, the corresponding stage in Git for Windows'
release process failed and required the stage definition to adapt the
`nuget.exe` invocation, and then the stage had to be re-deployed.
Without re-deploying _all_ stages, of course, and certainly without
having to modify already-tagged code and to re-build all artifacts. Yet
that is what this paradigm of GitHub Actions would have one do: whenever
any workflow definition needs to be adjusted, the changes need to be
committed into the target repository, a new tag has to be created, and
the entire release process has to be started again.

It goes without saying that Git for Windows has to work around this
limitation and reinstate the same flexibility that the Azure Release
Pipeline provided.

This commit is a step in that direction: it implements the `GitHub
Release` stage as a composite GitHub Action, with the idea that the
actual `release-git` workflow references this Action _in a different
branch_. That way, the `github-release` Action can be modified after a
`release-git` run failed, making the necessary changes in said different
branch, and then a re-run of the failed job will pickup those necessary
changes.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This stage performs the part of the Git for Windows release process
where after the GitHub Release is created, the corresponding commits in
the MINGW-packages and build-extra repositories are pushed to reflect
the new Git for Windows version.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This stage sends out the announcement mail.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This stage's responsibility is to update the Git for Windows home page
to point to the new version.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
As part of each Git for Windows release, the `mingw-w64-git` Pacman
packages are built and used to make the installers, MinGits, portable
Gits etc.

This here stage's purpose is to publish those Pacman packages (this can
only be done once the Git for Windows installer is validated manually
and released to the wild).

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
While it is really uncertain how many users there still are, Git for
Windows continues to publish these NuGet packages as part of each new
Git for Windows release.

This commit adds the equivalent of the Azure Release Pipeline's stage to
do that.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This is admittedly a bit tricky.

One of the _major_ reasons why Git for Windows stayed with Azure Release
Pipelines for _so long_ is that in reality, it is simply just wishful
thinking to believe that one can version a release pipeline together
with the code that is released.

In reality, things like Git for Windows versions are built from many,
many sources, and once the release artifacts are built, they are
validated manually (which is a bit tedious, but we owe that effort to
the millions of Git for Windows users out there). After that validation
happened, the artifacts are released. If that release process fails
(which happens quite frequently, typically due to changes outside of our
control, such as new ways to upload NuGet packages that require changes
in the release process), it is delusional to think that one can edit a
workflow `.yml` and then re-tag and force-push the original code for
which release artifacts had already been built and validated tediously.
It's just unrealistic to think that we can uphold a theoretically
beautiful design (version all the things in the same repository!) in
return for a ginormous maintenance burden.

No, the reality is that once release artifacts are built, the tag from
which they are built is set in stone. And when parts of the release
process fail, they have to be adjusted accordingly _and independently_.

In Azure Release Pipelines, this fact was recognized by not only
allowing to version the Release Pipeline definition independently from
the code whose artifacts are released. Even more astutely, Azure Release
Pipelines are prepared for the all-too-common scenario where a single
stage fails to deploy completely and then the definition for that stage
has to be modified, to skip already-deployed parts, _just for that
release_, and the stage has to be re-deployed. And only that stage.

Unfortunately, there is still not really any adequate equivalent in
GitHub Actions.

The closest we can get is to:

- maintain the release workflow in a separate repository (which we
  hereby do, as the code whose build artifacts are deployed lives in
  git-for-windows/git while the release workflow lives in
  git-for-windows/git-for-windows-automation and we are only able to
  maintain the connection via a Check Run in the former repository by
  jumping through hoops and implementing a custom GitHub Action to
  "mirror" a Check Run into a different repository than the actual
  workflow run is attached to)

- even more importantly, define the individual stages of the release
  workflow in the form of composite Actions that are maintained in a
  _separate_ branch (called `release`). That allows us to edit and
  re-deploy any failing stage individually, without having to re-run the
  entire workflow: by editing the composite Action, push to said branch,
  and then re-run the failed job.

So that's what we do: we create a new `release-git` workflow. But the
actual logic for the jobs defined in that workflow is not even
maintained in the `main` branch, but instead in the `release` branch,
which will be liberally modified as needed whenever releases fail.

The idea is for the `release` branch to be identical to the `main`
branch whenever things go smoothly. And when they don't, the `release`
branch is where the action happens, and once the release worked, the
`release` branch is merged into `main` (fast-forwarding, usually,
otherwise the result will be pushed back to `release`) so that the
identical state of both branches is once again achieved.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
We want to imitate Azure Release Pipelines' feature where failed stages
can be edited and then re-deployed _without_ requiring the entire
workflow to be re-started anew when its definition was edited.

To that end, the workflow references composite Actions in the `release`
branch (which is expected to be initially identical to the 'main'
branch). That way, the definitions can be edited and pushed to that
`release` branch, and after that re-running failed jobs will pick up
those fixes.

Let's make sure that we do start with the `release` branch's head commit
being the same as the one the workflow run was started from.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
@dscho
Copy link
Member Author

dscho commented Mar 7, 2023

Okay, let's go.

@dscho dscho merged commit 69b0b5f into main Mar 7, 2023
@dscho dscho deleted the release branch March 7, 2023 20:53
@dscho
Copy link
Member Author

dscho commented Mar 8, 2023

Full disclosure: these are the fixes that were needed after merging this PR: 69b0b5f...f2ef370

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Port the Azure Release Pipeline to publish Git for Windows versions

2 participants