Adjust the tag-git workflow and create a release-git workflow to build Git for Windows fully using GitHub Actions#38
Merged
Conversation
178c716 to
8dfbfb8
Compare
51c242a to
b11caa3
Compare
One of the main reasons of the existence of the `git-for-windows-automation` repository is the functionality where the `build-and-deploy` workflow "mirrors" a Check Run into a repository that is different from the one that actually runs the workflow. This allows us to version the workflow definition implementing automated builds independently of the code that is actually being built. We will also use this functionality in the `git-artifacts` workflow (and related workflows). So let's make the code as reusable as possible by creating a new GitHub Action specifically for the purpose. To make things a bit more convenient, the Action maintains a state in a file in `runner.temp`. This file contains, among other things, the access token as well as the information when it expires. Since that is sensitive information and should not be accessible by the build processes that are run as part of the rest of the workflow, the state file is encrypted (using the GitHub App's private key that we need to generate an access token, anyway). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
f4b45a7 to
dfa7845
Compare
This is needed e.g. to download workflow jobs' logs or artifacts. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This makes it easier to organize and encapsulate the functionality. This commit is best viewed with `--color-moved`. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This should make the code both easier to read as well as to modify. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Git for Windows publishes two NuGet packages (x86_64 only) as part of its release process. Which means that the `git-artifacts` workflow must generate them. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
…cessary As part of the regular `makepkg-mingw` run, we already have a bare clone in `/usr/src/MINGW-packages/mingw-w64-git/git` in addition to the checkout in `/usr/src/MINGW-packages/mingw-w64-git/src/git`. No need to have yet another clone. We can operate on the checkout directly. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
These checksums are needed to complete the text of the announcement mail as well as the text of the GitHub Release. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
We need this because we want to know what is the target commit as early as possible so that we can mirror a Check Run to that target commit. This commit is best viewed with `--color-moved`. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The `actions/checkout` Action is a more canonical way to do things, and it also benefits from better performance (shallow by default, clones only the main branch, etc). While at it, check out the repository in the top-level workspace (we are no longer using that location for the `git-for-windows/git` checkout). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Just like we do in the `build-and-deploy` and the `tag-git` workflows, we now also mirror a Check Run to `git-for-windows/git` when running the `git-artifacts` workflow. Since we want to have one workflow run per architecture, we include the architecture in the Check Run name, so that all of them show up in the corresponding git-for-windows/git PR in the Checks tab. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The entire idea of caching the package, once built, is to avoid having to rebuild the Pacman package just because some _other_ step in the workflow failed. But using `actions/cache` as-is won't actually cache the artifact properly (or worse, it can cache it incompletely when the _build_ failed). Let's be more careful by using the combination of `actions/cache/save` and `actions/cache/restore`. While at it, skip other steps, too, when they are not needed because a cached package was restored. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Naturally, when the version is re-tagged, we have to rebuild the package... But previously, we did not, because the `tag-git` workflow run ID did not play any role in the decision whether to restore or rebuild. Now it does. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
…t` ran The `tag-git` workflow, on which the `git-artifacts` runs for the various architectures rely, publishes a Git bundle to update the release notes. To make use of that, the `git-artifacts` workflow needs to check out the `build-extra` repository and merge those changes. However, it _does_ happen from time to time that the main branch of `build-extra` advances between the `tag-git` and the `git-artifacts` run. Therefore, we need to tell Git to make a merge commit if necessary. Which means that we have to have Git committer information ready to go, and therefore we configure `user.*` in the `artifacts` job when building an `installer`, too. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The `-busybox` infix is not automatically inserted. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
If building from a tag that was generated in a given `tag-git` workflow run, we must first make sure that that workflow run finished and succeeded. This is particularly important because the plan is to teach the GitForWindowsHelper GitHub App to no longer trigger the `Git Artifacts` Azure Pipeline upon `/git-artifacts`, but instead to trigger a `tag-git` workflow run, wait for it to finish, and then to trigger the `git-artifacts` workflow runs (one for each architecture supported by Git for Windows). And the logic to wait for the workflow run to finish and then to trigger the `git-artifacts` workflow runs is a bit tricky: the actual `tag-git` workflow run does not provide the necessary metadata (the git-for-windows/git commit to which to mirror the Check Run) that would be accessible by the regular `get-a-workflow-run` GitHub REST API call. Therefore, the plan is not actually to wait for the `tag-git` _workflow run_ in `git-for-windows-automation` to complete, but instead for the `tag-git` _Check Run_ that is mirrored into `git-for-windows/git`. This Check Run is created and updated explicitly and therefore we could make sure that the relevant information is included there (the `output.text` attribute contains a link to the workflow run, including its ID, and the `head_sha` attribute points to the git-for-windows/git commit to which the newly generated tag points). Since that Check Run is marked as completed by the workflow run that is obviously still running, the webhook event might arrive, and the `git-artifacts` workflow runs might be triggered, _before_ the `tag-git` workflow run actually finished. Hence the need to potentially await its completion. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
If we ever need to mirror Check Runs into _multiple_ repositories, this change will allow that: by specifying a state file for a different repo, it is possible to maintain multiple states at the same time. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
When an entire file needs to be uploaded, it is probably not a good idea to read the contents into memory and pass them as parameter. It is much better to open a stream and pipe it through. To support that, we allow to pass a file read stream instead of a string or an object. Sadly, we cannot use arbitrary read stream, it has to be a file read stream, because we need to know the file name and the file length. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The Git for Windows project previously used an Azure Release Pipeline to publish new versions. This was very convenient, as Azure Release Pipelines not only allow versioning the Pipeline definition independently of the code that is being deployed, it allows the individual _stages_ to be versioned independently and even _while deploying_. This comes in handy when changes outside of Git for Windows' control require changes in the stage definition. For example, when the NuGet deployment process changed, the corresponding stage in Git for Windows' release process failed and required the stage definition to adapt the `nuget.exe` invocation, and then the stage had to be re-deployed. Without re-deploying _all_ stages, of course, and certainly without having to modify already-tagged code and to re-build all artifacts. Yet that is what this paradigm of GitHub Actions would have one do: whenever any workflow definition needs to be adjusted, the changes need to be committed into the target repository, a new tag has to be created, and the entire release process has to be started again. It goes without saying that Git for Windows has to work around this limitation and reinstate the same flexibility that the Azure Release Pipeline provided. This commit is a step in that direction: it implements the `GitHub Release` stage as a composite GitHub Action, with the idea that the actual `release-git` workflow references this Action _in a different branch_. That way, the `github-release` Action can be modified after a `release-git` run failed, making the necessary changes in said different branch, and then a re-run of the failed job will pickup those necessary changes. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This stage performs the part of the Git for Windows release process where after the GitHub Release is created, the corresponding commits in the MINGW-packages and build-extra repositories are pushed to reflect the new Git for Windows version. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This stage sends out the announcement mail. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This stage's responsibility is to update the Git for Windows home page to point to the new version. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
As part of each Git for Windows release, the `mingw-w64-git` Pacman packages are built and used to make the installers, MinGits, portable Gits etc. This here stage's purpose is to publish those Pacman packages (this can only be done once the Git for Windows installer is validated manually and released to the wild). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
While it is really uncertain how many users there still are, Git for Windows continues to publish these NuGet packages as part of each new Git for Windows release. This commit adds the equivalent of the Azure Release Pipeline's stage to do that. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This is admittedly a bit tricky. One of the _major_ reasons why Git for Windows stayed with Azure Release Pipelines for _so long_ is that in reality, it is simply just wishful thinking to believe that one can version a release pipeline together with the code that is released. In reality, things like Git for Windows versions are built from many, many sources, and once the release artifacts are built, they are validated manually (which is a bit tedious, but we owe that effort to the millions of Git for Windows users out there). After that validation happened, the artifacts are released. If that release process fails (which happens quite frequently, typically due to changes outside of our control, such as new ways to upload NuGet packages that require changes in the release process), it is delusional to think that one can edit a workflow `.yml` and then re-tag and force-push the original code for which release artifacts had already been built and validated tediously. It's just unrealistic to think that we can uphold a theoretically beautiful design (version all the things in the same repository!) in return for a ginormous maintenance burden. No, the reality is that once release artifacts are built, the tag from which they are built is set in stone. And when parts of the release process fail, they have to be adjusted accordingly _and independently_. In Azure Release Pipelines, this fact was recognized by not only allowing to version the Release Pipeline definition independently from the code whose artifacts are released. Even more astutely, Azure Release Pipelines are prepared for the all-too-common scenario where a single stage fails to deploy completely and then the definition for that stage has to be modified, to skip already-deployed parts, _just for that release_, and the stage has to be re-deployed. And only that stage. Unfortunately, there is still not really any adequate equivalent in GitHub Actions. The closest we can get is to: - maintain the release workflow in a separate repository (which we hereby do, as the code whose build artifacts are deployed lives in git-for-windows/git while the release workflow lives in git-for-windows/git-for-windows-automation and we are only able to maintain the connection via a Check Run in the former repository by jumping through hoops and implementing a custom GitHub Action to "mirror" a Check Run into a different repository than the actual workflow run is attached to) - even more importantly, define the individual stages of the release workflow in the form of composite Actions that are maintained in a _separate_ branch (called `release`). That allows us to edit and re-deploy any failing stage individually, without having to re-run the entire workflow: by editing the composite Action, push to said branch, and then re-run the failed job. So that's what we do: we create a new `release-git` workflow. But the actual logic for the jobs defined in that workflow is not even maintained in the `main` branch, but instead in the `release` branch, which will be liberally modified as needed whenever releases fail. The idea is for the `release` branch to be identical to the `main` branch whenever things go smoothly. And when they don't, the `release` branch is where the action happens, and once the release worked, the `release` branch is merged into `main` (fast-forwarding, usually, otherwise the result will be pushed back to `release`) so that the identical state of both branches is once again achieved. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
We want to imitate Azure Release Pipelines' feature where failed stages can be edited and then re-deployed _without_ requiring the entire workflow to be re-started anew when its definition was edited. To that end, the workflow references composite Actions in the `release` branch (which is expected to be initially identical to the 'main' branch). That way, the definitions can be edited and pushed to that `release` branch, and after that re-running failed jobs will pick up those fixes. Let's make sure that we do start with the `release` branch's head commit being the same as the one the workflow run was started from. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Member
Author
|
Okay, let's go. |
Member
Author
|
Full disclosure: these are the fixes that were needed after merging this PR: 69b0b5f...f2ef370 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
tag-gitworkflow so that it can tag full Git for Windows versions, not just snapshotsgit-artifactsworkflow to mirror Check Runs togit-for-windows/git(needs to encode the architecture as part of the Check Run name to avoid them overwriting each other)release-gitThis addresses #37