Skip to content

Commit 65028a1

Browse files
donoghucdonoghuc
committed
Use docker manifest for multi-architecture builds
This commit refactors the POC pipeline for pushing observabilty SRE containers to handle conflicts for tags based on target architectures. Cells with respective architectures build containers and push to the container registry with a unique identifier. Once those exist we introduce a separate step to use the docker manifest command to annotate those images such that a container client can download the correct image based on architecture. As a result for every artifact there will be 2 images pushed (one for each arch) and N manifests pushed. The manifests will handle the final naming that the consumer would expect.
1 parent 976f104 commit 65028a1

3 files changed

Lines changed: 73 additions & 10 deletions

File tree

‎.buildkite/scripts/dra/build-and-push-observability-sre.sh‎

Lines changed: 6 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,10 @@
11
#!/bin/bash
22
# Script to build and publish ObservabilitySRE container
3+
# Currently this is built on a host with the target architecture.
4+
# This allows us to utilize the make file for building the container and
5+
# to ensure the best compatability with the host architecture.
6+
# A later step in CI will take care of pushing a tag that references the right
7+
# image using `docker manifest` commands.
38

49
echo "Setting up environment"
510
source .buildkite/scripts/common/vm-agent.sh
@@ -19,16 +24,7 @@ ARCH_TAG="${ARCH:-x86_64}" # Default to x86_64 if ARCH is not set
1924

2025
echo "Architecture: ${ARCH_TAG}"
2126

22-
if [[ "${WORKFLOW_TYPE}" == "staging" ]]; then
23-
# For staging builds, Push the original qualified version with architecture
24-
# Ex: docker.elastic.co/logstash/logstash-observability-sre:8.19.0-aarch64
25-
ARCH_VERSION_TAG="${QUALIFIED_VERSION}-${ARCH_TAG}"
26-
echo "Tagging and pushing: ${REGISTRY_PATH}:${QUALIFIED_VERSION} as ${REGISTRY_PATH}:${ARCH_VERSION_TAG}"
27-
docker tag ${REGISTRY_PATH}:${QUALIFIED_VERSION} ${REGISTRY_PATH}:${ARCH_VERSION_TAG}
28-
docker push ${REGISTRY_PATH}:${ARCH_VERSION_TAG}
29-
fi
30-
31-
# For both staging and snapshot builds push the qualified version + the sha + architecture
27+
# Push a unique tag (version + SHA) for the current build WITH the architecture in the name
3228
# Ex: docker.elastic.co/logstash/logstash-observability-sre:8.19.0-SNAPSHOT-297226b1df-aarch64
3329
SHA_ARCH_TAG="${QUALIFIED_VERSION}-${SHA}-${ARCH_TAG}"
3430
echo "Tagging and pushing: ${REGISTRY_PATH}:${QUALIFIED_VERSION} as ${REGISTRY_PATH}:${SHA_ARCH_TAG}"

‎.buildkite/scripts/dra/generatesteps.py‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -118,6 +118,21 @@ def ship_observability_sre_image_steps(branch, workflow_type):
118118
export ARCH="x86_64"
119119
eval "$(rbenv init -)"
120120
.buildkite/scripts/dra/build-and-push-observability-sre.sh
121+
- label: ":docker: Create & Push ObservabilitySRE Multi-Arch Manifest / {branch}-{workflow_type.upper()}"
122+
key: "logstash_create_observability_sre_manifest"
123+
depends_on:
124+
- "logstash_build_and_ship_observability_sre_aarch64"
125+
- "logstash_build_and_ship_observability_sre_x86_64"
126+
agents:
127+
provider: gcp
128+
imageProject: elastic-images-prod
129+
image: family/platform-ingest-logstash-ubuntu-2204
130+
machineType: "n2-standard-8"
131+
command: |
132+
export WORKFLOW_TYPE="{workflow_type}"
133+
export PATH="/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH"
134+
eval "$(rbenv init -)"
135+
.buildkite/scripts/dra/multi-architecture-observability-sre.sh
121136
'''
122137
return step
123138

‎.buildkite/scripts/dra/multi-architecture-observability-sre.sh‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
#!/bin/bash
2+
# Script to create and push Docker manifest for multi-architecture support
3+
# This MUST be fun after build-and-push-observabilty-sre.sh!
4+
5+
source .buildkite/scripts/common/vm-agent.sh
6+
source .buildkite/scripts/dra/docker-env-setup.sh
7+
8+
docker_login
9+
QUALIFIED_VERSION="$(.buildkite/scripts/common/qualified-version.sh)"
10+
SHA="$(git rev-parse --short HEAD)"
11+
REGISTRY_PATH=docker.elastic.co/logstash/logstash-observability-sre
12+
13+
# Architecture-specific tags (created by the build steps)
14+
X86_64_TAG="${QUALIFIED_VERSION}-${SHA}-x86_64"
15+
AARCH64_TAG="${QUALIFIED_VERSION}-${SHA}-aarch64"
16+
17+
# Target manifest tags
18+
SHA_MANIFEST_TAG="${QUALIFIED_VERSION}-${SHA}"
19+
VERSION_MANIFEST_TAG="${QUALIFIED_VERSION}"
20+
21+
# Create and push manifest with SHA
22+
echo "Creating manifest list for: ${REGISTRY_PATH}:${SHA_MANIFEST_TAG}"
23+
docker manifest create ${REGISTRY_PATH}:${SHA_MANIFEST_TAG} \
24+
${REGISTRY_PATH}:${X86_64_TAG} \
25+
${REGISTRY_PATH}:${AARCH64_TAG}
26+
27+
docker manifest annotate ${REGISTRY_PATH}:${SHA_MANIFEST_TAG} \
28+
${REGISTRY_PATH}:${X86_64_TAG} --os linux --arch amd64
29+
30+
docker manifest annotate ${REGISTRY_PATH}:${SHA_MANIFEST_TAG} \
31+
${REGISTRY_PATH}:${AARCH64_TAG} --os linux --arch arm64
32+
33+
echo "Pushing manifest: ${REGISTRY_PATH}:${SHA_MANIFEST_TAG}"
34+
docker manifest push ${REGISTRY_PATH}:${SHA_MANIFEST_TAG}
35+
36+
# Create and push manifest without SHA (just version)
37+
echo "Creating manifest list for: ${REGISTRY_PATH}:${VERSION_MANIFEST_TAG}"
38+
docker manifest create ${REGISTRY_PATH}:${VERSION_MANIFEST_TAG} \
39+
${REGISTRY_PATH}:${X86_64_TAG} \
40+
${REGISTRY_PATH}:${AARCH64_TAG}
41+
42+
docker manifest annotate ${REGISTRY_PATH}:${VERSION_MANIFEST_TAG} \
43+
${REGISTRY_PATH}:${X86_64_TAG} --os linux --arch amd64
44+
45+
docker manifest annotate ${REGISTRY_PATH}:${VERSION_MANIFEST_TAG} \
46+
${REGISTRY_PATH}:${AARCH64_TAG} --os linux --arch arm64
47+
48+
echo "Pushing manifest: ${REGISTRY_PATH}:${VERSION_MANIFEST_TAG}"
49+
docker manifest push ${REGISTRY_PATH}:${VERSION_MANIFEST_TAG}
50+
51+
# Teardown Docker environment
52+
source .buildkite/scripts/dra/docker-env-teardown.sh

0 commit comments

Comments
 (0)