Skip to content

feat: support building integrations in arm#25195

Closed
mdelapenya wants to merge 8 commits intoelastic:mainfrom
mdelapenya:build-integrations-multi-platform
Closed

feat: support building integrations in arm#25195
mdelapenya wants to merge 8 commits intoelastic:mainfrom
mdelapenya:build-integrations-multi-platform

Conversation

@mdelapenya
Copy link
Copy Markdown
Contributor

@mdelapenya mdelapenya commented Apr 21, 2021

What does this PR do?

It extracts the logic to build&push integrations' docker images using mage to a method in the pipeline, that will be used to be run in parallel: one branch for amd and one for arm. The result is having the integrations built in ARM format.

Besides that, we are adding a new mage command manifesttool:pushSupportedVersions for pushing the integration images with multi-platform support. We are using a tool provided by Infra to perform that task, which we wrapped into the new mage command. This mage command:

  • will be run only on linux machines, because is coupled with the CI workers. That could be changed for new images.
  • it will calculate the Docker tag used on the compose:buildSupportedVersions reading the values from the supported-versions.yml file for each integration module. It will add logic to build the proper tag based in the ${MODULE}_VARIANT and ${MODULE}_VERSION, and for the specific use case of ceph we are adding an ${MODULE}_INDEX to control the custom image number we use internally for tracking changes.

Why is it important?

The integrations are built for AMD only, and the e2e testing framework needs to consume them in ARM to provide a full experience testing in ARM.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • Do we need to use an agent in the first stage? Instead, I think we can move that agent to the checkout stage

Related issues

Other considerations

Maybe this approach could be discarded if we realise that maintaining ARM/AMD images for the integrations is overkill. It could be the case where we have to build and maintain the binaries for each integration in both platforms, very likely because there is no ARM flavour of the image. Because the existence of ARM images for the integrations is blocking elastic/e2e-testing#707, maybe we can get to the point of not running e2e tests for the integrations.

@mdelapenya mdelapenya self-assigned this Apr 21, 2021
@mdelapenya mdelapenya added the Team:Automation Label for the Observability productivity team label Apr 21, 2021
@mdelapenya mdelapenya requested a review from a team April 21, 2021 09:52
@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Apr 21, 2021
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Apr 21, 2021

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Reason: null

  • Start Time: 2022-02-03T10:19:02.712+0000

  • Duration: 151 min 10 sec

  • Commit: 62518f5

Test stats 🧪

Test Results
Failed 0
Passed 47607
Skipped 4281
Total 51888

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@mdelapenya mdelapenya requested a review from jsoriano April 23, 2021 18:23
mdelapenya added a commit to mdelapenya/e2e-testing that referenced this pull request Apr 23, 2021
Comment on lines +60 to +61
parallelTasks["arm_metricbeat"] = generateBuildStep(platform: "arm", beatPath: 'metricbeat')
parallelTasks["ubuntu_20_metricbeat"] = generateBuildStep(platform: "ubuntu-20", beatPath: 'metricbeat')
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are arm and ubuntu-20 jenkins labels? It looks weird to me to see an OS and an architecture used for the same platform parameter.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, they are Jenkins labels

case moduleUppercase + "_CODENAME":
case moduleUppercase + "_VARIANT":
codename = v
case moduleUppercase + "_INDEX":
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a significative change, not sure if we want to do it this way.

So far, for good or bad, the variables set in each variant don't have any special meaning, they are just applied when running docker-compose subcommands. Currently we know that when we want to do something with all the variants, we have to call docker-compose once for each variant, passing the variables defined there.

With this change we are forcing a convention that may affect what images are used and which ones are cached. For example we have images with index different to 1 (postgresql, nats, prometheus...). Should we also define a _INDEX variable apart of defining it in the docker-compose.yml file? This can be error-prone.

If this is done to cover the ceph images, I would prefer to rethink how ceph is tested, or even reduce its coverage, before introducing this logic here.

In see though that this system complicates the obtention of the tags. Maybe another approach for that is to parse the docker-compose.yml files to get the image fields for each service, and replace the variables there for each one of the variants. Imitating this logic of docker-compose can be a bit hacky, but it is a well known format, so it can be still clean. And this way we could continue having any variable in these files without expecting any special meaning.

mdelapenya added a commit to elastic/e2e-testing that referenced this pull request Jun 1, 2021
* feat: add support for running the tests in ARM

* fix: arm workers are not immutable yet

* chore: calculate artifact from the GOARCH variable

At this moment we support amd64 and arm64

* chore: run elasticsearch in the right platform

* chore: add log for architecture

* chore: print env variable for GOARCH

* chore: set GOARCH based in the worker

* fix: wrong copy&paste

* chore: install k8s dependencies on ARM

* chore: install godog using GOARCH

* fix: proper values for GO env in Makefile

* chore: add log when requesting a node

* fix: define Groovy variable

* chore: use proper platform for kibana

* chore: simplify platforms selection logic

* fix: use proper CI label for node selector

* chore: remove running containers in clean-docker script

* chore: clean up docker state on mutable workers

* chore: use parallel branch name for filebeat logs

* fix: add forgotten double quotes

* chore: print Docker compose version On ARM

* chore: use aarch64 architecture for RPM packages

* chore: support ARM in tar installer

* fix: proper arch for TAR installer

* chore: log when the artifact is found

* chore: be defensive about no running containers

* chore: add tests for RPM artifact name

* chore: add tests for x86_64 TAR artifact

* chore: remove docker label from CI workers

* chore: separate ARM tests in a second parallel stage

This pretends to avoid using too many workers, adding a second layer for ARM.

* chore: update stage name

* fix: update arm target platform

* chore: use GOARCH in the junit file

* chore: use GOARCH in filebeat logs

* chore: run all stages in parallel

* chore: add platform to log when skipping tests

* fix: use a filebeat version that supports multiplatform

* chore: do not run integrations on ARM

See elastic/beats#25195

* chore: move test resources for ARM

* chore: use staging environment for package-registry

See https://github.com/elastic/package-storage

* fix: remove clean because arm workers are now immutable

* chore: use new multiplatform images

* chore: use new multiplatform base images for fleet-server

* chore: add platform to elastic-agent cloud flavour

* fix: use proper variable for platform

* chore: do not clean up installer's workspace

Because we are using ephemeral workers, we do not want to remove previous files

* Revert "chore: do not clean up installer's workspace"

This reverts commit 9104501.

* chore(ci): skip pull of images from code

* Revert "chore(ci): skip pull of images from code"

This reverts commit 01cc441.

* chore: set platform when pulling images

* chore: update packer cache with our systemd images

* fix: request proper platform image for the agent

* feat: exclude scenarios per platform

* chore: remove tags for endpoint, as they are not used at all

They are also blocking the addition of a parent tag at the feature level, as maximun number of tags is 3

* fix: skip endpoint integration on ARM64

* feat: provide supported platforms at the scenario level on CI

The scenario will be able to override its parent suite's platforms, making
possible to select different platforms at the scenario level.

* fix: use proper length method for groovy arrays

* fix: groovy arrays use size

* fix: override suite platforms per scenario

* chore: update Jenkins label for arm workers

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

* chore: simplify platforms, reusing stack

* chore: explicitly pass GOARCH to CI script

* Revert "chore: update Jenkins label for arm workers"

This reverts commit 9ef81ca.

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
mergify bot pushed a commit to elastic/e2e-testing that referenced this pull request Jun 1, 2021
* feat: add support for running the tests in ARM

* fix: arm workers are not immutable yet

* chore: calculate artifact from the GOARCH variable

At this moment we support amd64 and arm64

* chore: run elasticsearch in the right platform

* chore: add log for architecture

* chore: print env variable for GOARCH

* chore: set GOARCH based in the worker

* fix: wrong copy&paste

* chore: install k8s dependencies on ARM

* chore: install godog using GOARCH

* fix: proper values for GO env in Makefile

* chore: add log when requesting a node

* fix: define Groovy variable

* chore: use proper platform for kibana

* chore: simplify platforms selection logic

* fix: use proper CI label for node selector

* chore: remove running containers in clean-docker script

* chore: clean up docker state on mutable workers

* chore: use parallel branch name for filebeat logs

* fix: add forgotten double quotes

* chore: print Docker compose version On ARM

* chore: use aarch64 architecture for RPM packages

* chore: support ARM in tar installer

* fix: proper arch for TAR installer

* chore: log when the artifact is found

* chore: be defensive about no running containers

* chore: add tests for RPM artifact name

* chore: add tests for x86_64 TAR artifact

* chore: remove docker label from CI workers

* chore: separate ARM tests in a second parallel stage

This pretends to avoid using too many workers, adding a second layer for ARM.

* chore: update stage name

* fix: update arm target platform

* chore: use GOARCH in the junit file

* chore: use GOARCH in filebeat logs

* chore: run all stages in parallel

* chore: add platform to log when skipping tests

* fix: use a filebeat version that supports multiplatform

* chore: do not run integrations on ARM

See elastic/beats#25195

* chore: move test resources for ARM

* chore: use staging environment for package-registry

See https://github.com/elastic/package-storage

* fix: remove clean because arm workers are now immutable

* chore: use new multiplatform images

* chore: use new multiplatform base images for fleet-server

* chore: add platform to elastic-agent cloud flavour

* fix: use proper variable for platform

* chore: do not clean up installer's workspace

Because we are using ephemeral workers, we do not want to remove previous files

* Revert "chore: do not clean up installer's workspace"

This reverts commit 9104501.

* chore(ci): skip pull of images from code

* Revert "chore(ci): skip pull of images from code"

This reverts commit 01cc441.

* chore: set platform when pulling images

* chore: update packer cache with our systemd images

* fix: request proper platform image for the agent

* feat: exclude scenarios per platform

* chore: remove tags for endpoint, as they are not used at all

They are also blocking the addition of a parent tag at the feature level, as maximun number of tags is 3

* fix: skip endpoint integration on ARM64

* feat: provide supported platforms at the scenario level on CI

The scenario will be able to override its parent suite's platforms, making
possible to select different platforms at the scenario level.

* fix: use proper length method for groovy arrays

* fix: groovy arrays use size

* fix: override suite platforms per scenario

* chore: update Jenkins label for arm workers

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

* chore: simplify platforms, reusing stack

* chore: explicitly pass GOARCH to CI script

* Revert "chore: update Jenkins label for arm workers"

This reverts commit 9ef81ca.

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
(cherry picked from commit e1a97f3)

# Conflicts:
#	.ci/packer_cache.sh
#	cli/config/compose/profiles/fleet/docker-compose.yml
#	cli/config/compose/profiles/metricbeat/docker-compose.yml
#	e2e/_suites/fleet/stand-alone.go
#	internal/deploy/docker_client.go
#	internal/installer/elasticagent_deb.go
#	internal/installer/elasticagent_docker.go
#	internal/installer/elasticagent_rpm.go
#	internal/installer/elasticagent_tar.go
#	internal/utils/utils_test.go
mergify bot pushed a commit to elastic/e2e-testing that referenced this pull request Jun 1, 2021
* feat: add support for running the tests in ARM

* fix: arm workers are not immutable yet

* chore: calculate artifact from the GOARCH variable

At this moment we support amd64 and arm64

* chore: run elasticsearch in the right platform

* chore: add log for architecture

* chore: print env variable for GOARCH

* chore: set GOARCH based in the worker

* fix: wrong copy&paste

* chore: install k8s dependencies on ARM

* chore: install godog using GOARCH

* fix: proper values for GO env in Makefile

* chore: add log when requesting a node

* fix: define Groovy variable

* chore: use proper platform for kibana

* chore: simplify platforms selection logic

* fix: use proper CI label for node selector

* chore: remove running containers in clean-docker script

* chore: clean up docker state on mutable workers

* chore: use parallel branch name for filebeat logs

* fix: add forgotten double quotes

* chore: print Docker compose version On ARM

* chore: use aarch64 architecture for RPM packages

* chore: support ARM in tar installer

* fix: proper arch for TAR installer

* chore: log when the artifact is found

* chore: be defensive about no running containers

* chore: add tests for RPM artifact name

* chore: add tests for x86_64 TAR artifact

* chore: remove docker label from CI workers

* chore: separate ARM tests in a second parallel stage

This pretends to avoid using too many workers, adding a second layer for ARM.

* chore: update stage name

* fix: update arm target platform

* chore: use GOARCH in the junit file

* chore: use GOARCH in filebeat logs

* chore: run all stages in parallel

* chore: add platform to log when skipping tests

* fix: use a filebeat version that supports multiplatform

* chore: do not run integrations on ARM

See elastic/beats#25195

* chore: move test resources for ARM

* chore: use staging environment for package-registry

See https://github.com/elastic/package-storage

* fix: remove clean because arm workers are now immutable

* chore: use new multiplatform images

* chore: use new multiplatform base images for fleet-server

* chore: add platform to elastic-agent cloud flavour

* fix: use proper variable for platform

* chore: do not clean up installer's workspace

Because we are using ephemeral workers, we do not want to remove previous files

* Revert "chore: do not clean up installer's workspace"

This reverts commit 9104501.

* chore(ci): skip pull of images from code

* Revert "chore(ci): skip pull of images from code"

This reverts commit 01cc441.

* chore: set platform when pulling images

* chore: update packer cache with our systemd images

* fix: request proper platform image for the agent

* feat: exclude scenarios per platform

* chore: remove tags for endpoint, as they are not used at all

They are also blocking the addition of a parent tag at the feature level, as maximun number of tags is 3

* fix: skip endpoint integration on ARM64

* feat: provide supported platforms at the scenario level on CI

The scenario will be able to override its parent suite's platforms, making
possible to select different platforms at the scenario level.

* fix: use proper length method for groovy arrays

* fix: groovy arrays use size

* fix: override suite platforms per scenario

* chore: update Jenkins label for arm workers

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

* chore: simplify platforms, reusing stack

* chore: explicitly pass GOARCH to CI script

* Revert "chore: update Jenkins label for arm workers"

This reverts commit 9ef81ca.

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
(cherry picked from commit e1a97f3)

# Conflicts:
#	cli/config/compose/profiles/fleet/docker-compose.yml
#	cli/config/compose/profiles/metricbeat/docker-compose.yml
#	e2e/_suites/fleet/stand-alone.go
#	internal/installer/elasticagent_deb.go
#	internal/installer/elasticagent_docker.go
#	internal/installer/elasticagent_rpm.go
#	internal/installer/elasticagent_tar.go
#	internal/utils/utils_test.go
mdelapenya added a commit to mdelapenya/e2e-testing that referenced this pull request Jun 1, 2021
* feat: add support for running the tests in ARM

* fix: arm workers are not immutable yet

* chore: calculate artifact from the GOARCH variable

At this moment we support amd64 and arm64

* chore: run elasticsearch in the right platform

* chore: add log for architecture

* chore: print env variable for GOARCH

* chore: set GOARCH based in the worker

* fix: wrong copy&paste

* chore: install k8s dependencies on ARM

* chore: install godog using GOARCH

* fix: proper values for GO env in Makefile

* chore: add log when requesting a node

* fix: define Groovy variable

* chore: use proper platform for kibana

* chore: simplify platforms selection logic

* fix: use proper CI label for node selector

* chore: remove running containers in clean-docker script

* chore: clean up docker state on mutable workers

* chore: use parallel branch name for filebeat logs

* fix: add forgotten double quotes

* chore: print Docker compose version On ARM

* chore: use aarch64 architecture for RPM packages

* chore: support ARM in tar installer

* fix: proper arch for TAR installer

* chore: log when the artifact is found

* chore: be defensive about no running containers

* chore: add tests for RPM artifact name

* chore: add tests for x86_64 TAR artifact

* chore: remove docker label from CI workers

* chore: separate ARM tests in a second parallel stage

This pretends to avoid using too many workers, adding a second layer for ARM.

* chore: update stage name

* fix: update arm target platform

* chore: use GOARCH in the junit file

* chore: use GOARCH in filebeat logs

* chore: run all stages in parallel

* chore: add platform to log when skipping tests

* fix: use a filebeat version that supports multiplatform

* chore: do not run integrations on ARM

See elastic/beats#25195

* chore: move test resources for ARM

* chore: use staging environment for package-registry

See https://github.com/elastic/package-storage

* fix: remove clean because arm workers are now immutable

* chore: use new multiplatform images

* chore: use new multiplatform base images for fleet-server

* chore: add platform to elastic-agent cloud flavour

* fix: use proper variable for platform

* chore: do not clean up installer's workspace

Because we are using ephemeral workers, we do not want to remove previous files

* Revert "chore: do not clean up installer's workspace"

This reverts commit 9104501.

* chore(ci): skip pull of images from code

* Revert "chore(ci): skip pull of images from code"

This reverts commit 01cc441.

* chore: set platform when pulling images

* chore: update packer cache with our systemd images

* fix: request proper platform image for the agent

* feat: exclude scenarios per platform

* chore: remove tags for endpoint, as they are not used at all

They are also blocking the addition of a parent tag at the feature level, as maximun number of tags is 3

* fix: skip endpoint integration on ARM64

* feat: provide supported platforms at the scenario level on CI

The scenario will be able to override its parent suite's platforms, making
possible to select different platforms at the scenario level.

* fix: use proper length method for groovy arrays

* fix: groovy arrays use size

* fix: override suite platforms per scenario

* chore: update Jenkins label for arm workers

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

* chore: simplify platforms, reusing stack

* chore: explicitly pass GOARCH to CI script

* Revert "chore: update Jenkins label for arm workers"

This reverts commit 9ef81ca.

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
mdelapenya added a commit to mdelapenya/e2e-testing that referenced this pull request Jun 1, 2021
* feat: add support for running the tests in ARM

* fix: arm workers are not immutable yet

* chore: calculate artifact from the GOARCH variable

At this moment we support amd64 and arm64

* chore: run elasticsearch in the right platform

* chore: add log for architecture

* chore: print env variable for GOARCH

* chore: set GOARCH based in the worker

* fix: wrong copy&paste

* chore: install k8s dependencies on ARM

* chore: install godog using GOARCH

* fix: proper values for GO env in Makefile

* chore: add log when requesting a node

* fix: define Groovy variable

* chore: use proper platform for kibana

* chore: simplify platforms selection logic

* fix: use proper CI label for node selector

* chore: remove running containers in clean-docker script

* chore: clean up docker state on mutable workers

* chore: use parallel branch name for filebeat logs

* fix: add forgotten double quotes

* chore: print Docker compose version On ARM

* chore: use aarch64 architecture for RPM packages

* chore: support ARM in tar installer

* fix: proper arch for TAR installer

* chore: log when the artifact is found

* chore: be defensive about no running containers

* chore: add tests for RPM artifact name

* chore: add tests for x86_64 TAR artifact

* chore: remove docker label from CI workers

* chore: separate ARM tests in a second parallel stage

This pretends to avoid using too many workers, adding a second layer for ARM.

* chore: update stage name

* fix: update arm target platform

* chore: use GOARCH in the junit file

* chore: use GOARCH in filebeat logs

* chore: run all stages in parallel

* chore: add platform to log when skipping tests

* fix: use a filebeat version that supports multiplatform

* chore: do not run integrations on ARM

See elastic/beats#25195

* chore: move test resources for ARM

* chore: use staging environment for package-registry

See https://github.com/elastic/package-storage

* fix: remove clean because arm workers are now immutable

* chore: use new multiplatform images

* chore: use new multiplatform base images for fleet-server

* chore: add platform to elastic-agent cloud flavour

* fix: use proper variable for platform

* chore: do not clean up installer's workspace

Because we are using ephemeral workers, we do not want to remove previous files

* Revert "chore: do not clean up installer's workspace"

This reverts commit 9104501.

* chore(ci): skip pull of images from code

* Revert "chore(ci): skip pull of images from code"

This reverts commit 01cc441.

* chore: set platform when pulling images

* chore: update packer cache with our systemd images

* fix: request proper platform image for the agent

* feat: exclude scenarios per platform

* chore: remove tags for endpoint, as they are not used at all

They are also blocking the addition of a parent tag at the feature level, as maximun number of tags is 3

* fix: skip endpoint integration on ARM64

* feat: provide supported platforms at the scenario level on CI

The scenario will be able to override its parent suite's platforms, making
possible to select different platforms at the scenario level.

* fix: use proper length method for groovy arrays

* fix: groovy arrays use size

* fix: override suite platforms per scenario

* chore: update Jenkins label for arm workers

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

* chore: simplify platforms, reusing stack

* chore: explicitly pass GOARCH to CI script

* Revert "chore: update Jenkins label for arm workers"

This reverts commit 9ef81ca.

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
mdelapenya added a commit to elastic/e2e-testing that referenced this pull request Jun 1, 2021
…1223)

* feat: add support for running the tests in ARM (#707)

* feat: add support for running the tests in ARM

* fix: arm workers are not immutable yet

* chore: calculate artifact from the GOARCH variable

At this moment we support amd64 and arm64

* chore: run elasticsearch in the right platform

* chore: add log for architecture

* chore: print env variable for GOARCH

* chore: set GOARCH based in the worker

* fix: wrong copy&paste

* chore: install k8s dependencies on ARM

* chore: install godog using GOARCH

* fix: proper values for GO env in Makefile

* chore: add log when requesting a node

* fix: define Groovy variable

* chore: use proper platform for kibana

* chore: simplify platforms selection logic

* fix: use proper CI label for node selector

* chore: remove running containers in clean-docker script

* chore: clean up docker state on mutable workers

* chore: use parallel branch name for filebeat logs

* fix: add forgotten double quotes

* chore: print Docker compose version On ARM

* chore: use aarch64 architecture for RPM packages

* chore: support ARM in tar installer

* fix: proper arch for TAR installer

* chore: log when the artifact is found

* chore: be defensive about no running containers

* chore: add tests for RPM artifact name

* chore: add tests for x86_64 TAR artifact

* chore: remove docker label from CI workers

* chore: separate ARM tests in a second parallel stage

This pretends to avoid using too many workers, adding a second layer for ARM.

* chore: update stage name

* fix: update arm target platform

* chore: use GOARCH in the junit file

* chore: use GOARCH in filebeat logs

* chore: run all stages in parallel

* chore: add platform to log when skipping tests

* fix: use a filebeat version that supports multiplatform

* chore: do not run integrations on ARM

See elastic/beats#25195

* chore: move test resources for ARM

* chore: use staging environment for package-registry

See https://github.com/elastic/package-storage

* fix: remove clean because arm workers are now immutable

* chore: use new multiplatform images

* chore: use new multiplatform base images for fleet-server

* chore: add platform to elastic-agent cloud flavour

* fix: use proper variable for platform

* chore: do not clean up installer's workspace

Because we are using ephemeral workers, we do not want to remove previous files

* Revert "chore: do not clean up installer's workspace"

This reverts commit 9104501.

* chore(ci): skip pull of images from code

* Revert "chore(ci): skip pull of images from code"

This reverts commit 01cc441.

* chore: set platform when pulling images

* chore: update packer cache with our systemd images

* fix: request proper platform image for the agent

* feat: exclude scenarios per platform

* chore: remove tags for endpoint, as they are not used at all

They are also blocking the addition of a parent tag at the feature level, as maximun number of tags is 3

* fix: skip endpoint integration on ARM64

* feat: provide supported platforms at the scenario level on CI

The scenario will be able to override its parent suite's platforms, making
possible to select different platforms at the scenario level.

* fix: use proper length method for groovy arrays

* fix: groovy arrays use size

* fix: override suite platforms per scenario

* chore: update Jenkins label for arm workers

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

* chore: simplify platforms, reusing stack

* chore: explicitly pass GOARCH to CI script

* Revert "chore: update Jenkins label for arm workers"

This reverts commit 9ef81ca.

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

* fix: remove packer_cache from maintenance branches

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
mdelapenya added a commit to elastic/e2e-testing that referenced this pull request Jun 1, 2021
#1224)

* feat: add support for running the tests in ARM (#707)

* feat: add support for running the tests in ARM

* fix: arm workers are not immutable yet

* chore: calculate artifact from the GOARCH variable

At this moment we support amd64 and arm64

* chore: run elasticsearch in the right platform

* chore: add log for architecture

* chore: print env variable for GOARCH

* chore: set GOARCH based in the worker

* fix: wrong copy&paste

* chore: install k8s dependencies on ARM

* chore: install godog using GOARCH

* fix: proper values for GO env in Makefile

* chore: add log when requesting a node

* fix: define Groovy variable

* chore: use proper platform for kibana

* chore: simplify platforms selection logic

* fix: use proper CI label for node selector

* chore: remove running containers in clean-docker script

* chore: clean up docker state on mutable workers

* chore: use parallel branch name for filebeat logs

* fix: add forgotten double quotes

* chore: print Docker compose version On ARM

* chore: use aarch64 architecture for RPM packages

* chore: support ARM in tar installer

* fix: proper arch for TAR installer

* chore: log when the artifact is found

* chore: be defensive about no running containers

* chore: add tests for RPM artifact name

* chore: add tests for x86_64 TAR artifact

* chore: remove docker label from CI workers

* chore: separate ARM tests in a second parallel stage

This pretends to avoid using too many workers, adding a second layer for ARM.

* chore: update stage name

* fix: update arm target platform

* chore: use GOARCH in the junit file

* chore: use GOARCH in filebeat logs

* chore: run all stages in parallel

* chore: add platform to log when skipping tests

* fix: use a filebeat version that supports multiplatform

* chore: do not run integrations on ARM

See elastic/beats#25195

* chore: move test resources for ARM

* chore: use staging environment for package-registry

See https://github.com/elastic/package-storage

* fix: remove clean because arm workers are now immutable

* chore: use new multiplatform images

* chore: use new multiplatform base images for fleet-server

* chore: add platform to elastic-agent cloud flavour

* fix: use proper variable for platform

* chore: do not clean up installer's workspace

Because we are using ephemeral workers, we do not want to remove previous files

* Revert "chore: do not clean up installer's workspace"

This reverts commit 9104501.

* chore(ci): skip pull of images from code

* Revert "chore(ci): skip pull of images from code"

This reverts commit 01cc441.

* chore: set platform when pulling images

* chore: update packer cache with our systemd images

* fix: request proper platform image for the agent

* feat: exclude scenarios per platform

* chore: remove tags for endpoint, as they are not used at all

They are also blocking the addition of a parent tag at the feature level, as maximun number of tags is 3

* fix: skip endpoint integration on ARM64

* feat: provide supported platforms at the scenario level on CI

The scenario will be able to override its parent suite's platforms, making
possible to select different platforms at the scenario level.

* fix: use proper length method for groovy arrays

* fix: groovy arrays use size

* fix: override suite platforms per scenario

* chore: update Jenkins label for arm workers

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

* chore: simplify platforms, reusing stack

* chore: explicitly pass GOARCH to CI script

* Revert "chore: update Jenkins label for arm workers"

This reverts commit 9ef81ca.

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

* chore: remove packer_cache from maintenance branch

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
@mergify
Copy link
Copy Markdown
Contributor

mergify bot commented Sep 22, 2021

This pull request does not have a backport label. Could you fix it @mdelapenya? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

@mergify mergify bot added the backport-skip Skip notification from the automated backport with mergify label Sep 22, 2021
@mergify
Copy link
Copy Markdown
Contributor

mergify bot commented Mar 16, 2022

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b build-integrations-multi-platform upstream/build-integrations-multi-platform
git merge upstream/main
git push upstream build-integrations-multi-platform

@mdelapenya
Copy link
Copy Markdown
Contributor Author

With the move to integration packages, I'd say this PR is not needed anymore, so we can close it

@jlind23 jlind23 closed this Mar 31, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-skip Skip notification from the automated backport with mergify Team:Automation Label for the Observability productivity team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants