Build infrastructure for tagging and publishing a release.#1972
Build infrastructure for tagging and publishing a release.#1972sebastienvas merged 33 commits intoistio:masterfrom
Conversation
| \"draft\": true, \ | ||
| \"prerelease\": true \ | ||
| }" > ${REQUEST_FILE} | ||
| curl -s -S -X POST -o ${RESPONSE_FILE} -H "Accept: application/vnd.github.v3+json" -H "Content-Type: application/json" \ |
There was a problem hiding this comment.
Maybe add a retry ? --retry 3
|
It feels to me that we are re-implementing githubctl in shell, which is a step back. Before I go any further in the review, is there ways we can update githubctl to deal with all of this. Shell is a pain to maintain, and we can create proper retry logic, and nice unit tests in goland. Plus the github API is fully implemented and usually up to date in go-github. |
sebastienvas
left a comment
There was a problem hiding this comment.
Please fix the comment that I added about renaming debian package and also using cat instead of echo when printing json file for better readibility
|
After the release is done, we will revisit all the scripts and try to build better libraries such that release code on istio is minimal. Istio release code, should basically be configuration, everything else should be handled by the release tooling. I am fine leaving this for now. |
|
/lgtm cancel //PR changed after LGTM, removing LGTM. @ldemailly @mattdelco @rkpagadala @sebastienvas |
|
/lgtm |
|
/approve |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: rkpagadala, sebastienvas Assign the PR to them by writing The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these OWNERS Files:You can indicate your approval by writing |
|
/lgtm cancel //PR changed after LGTM, removing LGTM. @ldemailly @mattdelco @rkpagadala @sebastienvas |
This change adds infrastructure for the tag and publish steps in the Release Process. Each of these are run via Cloud Builder so I've made changes so the files can share the implementation. The following is the high-level call pattern of how things typically execute:
The invoker (usually the "Release Process")
The caller (if they don't already have a source tree present) needs to do a "repo init" and "repo sync" and then run 1 (or more, though not typical) of:
The steps run to request a "build" (an actual build, a tag, or a publish) from Cloud Builder (CB):
For a build:
release/start_gcb_build.sh uses run_build (in release/gcb_build_lib.sh) with
release/cloud_build.template.json, which has CB run repo, cloud_builder.sh,
update_and_create_archives.sh, store_artifacts.sh.
For a tag:
release/start_gcb_tag.sh uses run_build (in release/gcb_build_lib.sh) with
release/cloud_tag.template.json, which has CB run repo, publish_release.sh (the steps for
publishing to gcs/gcr/docker/github are disabled).
For a publish:
release/start_gcb_publish.sh uses run_build (in release/gcb_build_lib.sh) with
release/cloud_publish.template.json, which has CB run repo, publish_release.sh (the step
for tagging on github is disabled).
Commons/shared to the above 3 build/tag/publish:
release/gcb_build_lib.sh: This is the common functionality for making a build request out of a template
file, requesting a build, and then optionally waiting for the build to complete.
release/json_parse_shared.sh: This file is the shared ugly code for trying to hackishly parse json in bash
using minimal outside dependencies.
Steps run inside Cloud Builder
For a build:
release/cloud_builder.sh: runs push-debian/push-docker/upload-istioctl,
renames *-release.deb files to .deb
release/update_and_create_archives.sh: This script calls install/updateVersion.sh and
create_release_archives.sh twice. The first time is to create a set of "testonly"
tar files that point to the staging location, the second time creates a set that
point to the release location (though the location won't exist until publish is run).
release/create_release_archives.sh: creates the tar files. I adjust the script to
now expect istioctl- under istioctl/ .
release/store_artifacts.sh: Uploads files to GCS/GCR. I adjusted the script
to gzip the docker tars.
For a tag or publish:
release/publish_release.sh:
This is the top-level script run in CB for doing a tag and/or a publish. I try to do certain tasks
in this script so that other scripts only need to work from a local directory (to help address
any concerns from files being dependent on google infrastructure). The script optionally copies
a github token file from GCS to a local temp directory, and then it then optionally performs a
decryption of a github token file (using kms in Google Cloud). The staged build is optionally
copied from GCS to a local directory). After this point the script mostly works out of the temp
directory. The script then does 1 or more of: copy files to GCS, push to GCR/docker, create a tag
on github (using release/create_tag_reference.sh), create a release on github (using
release/create_github_release.sh).
release/create_tag_reference.sh:
This script makes tags and references on github using the github REST API. Currently it tags only
istio proxy but can be later changed to tag all the repos once the build is eventually changed to
builds using what's in the repo manifest rather than what's in the bazel WORKSPACE file.
release/create_github_release.sh:
Creates a draft release on github using a github token [file] and a local directory that contains the
build artifacts. The caller specifies the repo and sha for the release (and also typically populates
the local directory from the GCS staging dir).
Changes to prior build behavior
docker tar files going under docker/), but now debian files are in /deb and istioctl-* are in
/istioctl .
release location then fewer, if any, things should be broken).
and /istioctl subdirs).
(I'm tempted to gzip the files as they're generated but I'd have to touch the various
push-docker scripts in istio and proxy repos), but halves the size of the files and makes
other steps go faster (offhand I can't say if the net time change is a gain or a loss).
a manual step in the 0.2 releases.
gcs/gcr/docker/github).
I've moved the functionality into a consolidated functionality in gcb_build_lib.sh and
json_parse_shared.sh.
I've currently implemented it to use the same encryption key as the docker file ("DockerHub").
to the debian images (previously these files were just uploaded) and now when you view the
draft they're clickable URLs (github doesn't like whitespace between the "text" ).
run via airflow). Previously there was build-launching script that used repo tool to
grab the build template. Now most of the build-launch script is obtained via repo tool and
that script now uses a template that was grabbed at the same time. So, the only thing
that now lives in the top-level is a few calls to repo tool and a call to the build-launching script.
others took them as a combined "bucket/subdir" parameter. I've updated the former to the
latter.