Backport 16907 to 8.17: Use --qualifier in release manager (#16907)#16941
Merged
dliappis merged 1 commit intoelastic:8.17from Jan 24, 2025
Merged
Backport 16907 to 8.17: Use --qualifier in release manager (#16907)#16941dliappis merged 1 commit intoelastic:8.17from
dliappis merged 1 commit intoelastic:8.17from
Conversation
This commit uses the new --qualifier parameter in the release manager for publishing dra artifacts. Additionally, simplifies the expected variables to rely on a simple `VERSION_QUALIFIER`. Snapshot builds are skipped when VERSION_QUALIFIER is set. Finally, for helping to test DRA PRs, we also allow passing the `DRA_BRANCH` option/env var to override BUILDKITE_BRANCH. Closes https://github.com/elastic/ingest-dev/issues/4856
Contributor
Author
Note for reviewersThis is identical to #16913, but targeted to 8.17 instead. Automation couldn't be used due to conflicts in the build_docker script (we still build ubi8 in this branch). Comparing this PR against 8.x$ TARGET_BRANCH="8.x"; for fname in .buildkite/dra_pipeline.yml .buildkite/scripts/dra/build_docker.sh .buildkite/scripts/dra/build_packages.sh .buildkite/scripts/dra/generatesteps.py .buildkite/scripts/dra/publish.sh; do echo ">>> Comparing $fname against $TARGET_BRANCH"; git diff $TARGET_BRANCH -- $fname ; done
>>> Comparing .buildkite/dra_pipeline.yml against 8.x
>>> Comparing .buildkite/scripts/dra/build_docker.sh against 8.x
>>> Comparing .buildkite/scripts/dra/build_packages.sh against 8.x
>>> Comparing .buildkite/scripts/dra/generatesteps.py against 8.x
>>> Comparing .buildkite/scripts/dra/publish.sh against 8.xComparing this PR against main$ TARGET_BRANCH="main"; for fname in .buildkite/dra_pipeline.yml .buildkite/scripts/dra/build_docker.sh .buildkite/scripts/dra/build_packages.sh .buildkite/scripts/dra/generatesteps.py .buildkite/scripts/dra/publish.sh; do echo ">>> Comparing $fname against $TARGET_BRANCH"; git diff $TARGET_BRANCH -- $fname ; done
>>> Comparing .buildkite/dra_pipeline.yml against main
>>> Comparing .buildkite/scripts/dra/build_docker.sh against main
diff --git a/.buildkite/scripts/dra/build_docker.sh b/.buildkite/scripts/dra/build_docker.sh
index 938922782..891a70ce2 100755
--- a/.buildkite/scripts/dra/build_docker.sh
+++ b/.buildkite/scripts/dra/build_docker.sh
@@ -26,6 +26,10 @@ rake artifact:docker_oss || error "artifact:docker_oss build failed."
rake artifact:docker_wolfi || error "artifact:docker_wolfi build failed."
rake artifact:dockerfiles || error "artifact:dockerfiles build failed."
+if [[ "$ARCH" != "aarch64" ]]; then
+ rake artifact:docker_ubi8 || error "artifact:docker_ubi8 build failed."
+fi
+
if [[ "$WORKFLOW_TYPE" == "staging" ]] && [[ -n "$VERSION_QUALIFIER" ]]; then
# Qualifier is passed from CI as optional field and specify the version postfix
# in case of alpha or beta releases for staging builds only:
@@ -48,6 +52,10 @@ for file in build/logstash-*; do shasum $file;done
info "Uploading DRA artifacts in buildkite's artifact store ..."
# Note the deb, rpm tar.gz AARCH64 files generated has already been loaded by the build_packages.sh
images="logstash logstash-oss logstash-wolfi"
+if [ "$ARCH" != "aarch64" ]; then
+ # No logstash-ubi8 for AARCH64
+ images="logstash logstash-oss logstash-wolfi logstash-ubi8"
+fi
for image in ${images}; do
buildkite-agent artifact upload "build/$image-${STACK_VERSION}-docker-image-${ARCH}.tar.gz"
done
@@ -55,7 +63,7 @@ done
# Upload 'docker-build-context.tar.gz' files only when build x86_64, otherwise they will be
# overwritten when building aarch64 (or viceversa).
if [ "$ARCH" != "aarch64" ]; then
- for image in logstash logstash-oss logstash-wolfi logstash-ironbank; do
+ for image in logstash logstash-oss logstash-wolfi logstash-ubi8 logstash-ironbank; do
buildkite-agent artifact upload "build/${image}-${STACK_VERSION}-docker-build-context.tar.gz"
done
fi
>>> Comparing .buildkite/scripts/dra/build_packages.sh against main
>>> Comparing .buildkite/scripts/dra/generatesteps.py against main
>>> Comparing .buildkite/scripts/dra/publish.sh against main
diff --git a/.buildkite/scripts/dra/publish.sh b/.buildkite/scripts/dra/publish.sh
index 09ef1a2ac..c56fbaedf 100755
--- a/.buildkite/scripts/dra/publish.sh
+++ b/.buildkite/scripts/dra/publish.sh
@@ -42,6 +42,13 @@ if [ "$RELEASE_VER" != "7.17" ]; then
:
fi
+# Deleting ubi8 for aarch64 for the time being. This image itself is not being built, and it is not expected
+# by the release manager.
+# See https://github.com/elastic/infra/blob/master/cd/release/release-manager/project-configs/8.5/logstash.gradle
+# for more details.
+# TODO filter it out when uploading artifacts instead
+rm -f build/logstash-ubi8-${STACK_VERSION}-docker-image-aarch64.tar.gz
+
info "Downloaded ARTIFACTS sha report"
for file in build/logstash-*; do shasum $file;done
|
|
💚 Build Succeeded
cc @dliappis |
Contributor
Author
|
Successful snapshot build using this PR: https://buildkite.com/elastic/logstash-dra-snapshot-pipeline/builds/2077 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
(cherry picked from commit 9385cfa)
Automated backport had conflicts in
.buildkite/scripts/dra/build_docker.shwhere before main we also build ubi8.Release notes
[rn:skip]
What does this PR do?
This commit uses the new --qualifier parameter in the release manager for publishing dra artifacts. Additionally, simplifies the expected variables to rely on a simple
VERSION_QUALIFIER.Finally, we skip snapshot builds when VERSION_QUALIFIER is set.
Why is it important/What is the impact to the user?
Enables prerelease staging builds
How to test this PR
To test via this PR supply the following BK options to the staging pipeline:
which results in a successful build -> https://buildkite.com/elastic/logstash-dra-staging-pipeline/builds/177
Once it's merged, manual staging builds need to be triggered using only
VERSION_QUALIFIER; docs will be updated for https://docs.elastic.dev/ingest-dev-docs/logstash/dra#usage-of-version_qualifier_opt via a follow up PR.Also tested a snapshot build to ensure nothing's broken (this just requires
DRA_BRANCH="main"as params), it's successful -> https://buildkite.com/elastic/logstash-dra-snapshot-pipeline/builds/2052Related issues
Closes https://github.com/elastic/ingest-dev/issues/4856
Screenshots