Skip to content
This repository was archived by the owner on Sep 17, 2024. It is now read-only.

[7.x](backport #707) feat: add support for running the tests in ARM#1219

Closed
mergify[bot] wants to merge 1 commit into7.xfrom
mergify/bp/7.x/pr-707
Closed

[7.x](backport #707) feat: add support for running the tests in ARM#1219
mergify[bot] wants to merge 1 commit into7.xfrom
mergify/bp/7.x/pr-707

Conversation

@mergify
Copy link
Copy Markdown
Contributor

@mergify mergify bot commented Jun 1, 2021

This is an automatic backport of pull request #707 done by Mergify.
Cherry-pick of e1a97f3 has failed:

On branch mergify/bp/7.x/pr-707
Your branch is up to date with 'origin/7.x'.

You are currently cherry-picking commit e1a97f3.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   .ci/.e2e-tests.yaml
	modified:   .ci/Jenkinsfile
	modified:   .ci/scripts/clean-docker.sh
	modified:   .ci/scripts/functional-test.sh
	modified:   .ci/scripts/install-helm-test-dependencies.sh
	modified:   cli/config/compose/profiles/fleet/configurations/kibana.config.yml
	modified:   cli/config/compose/services/elastic-agent/centos/docker-compose.yml
	modified:   cli/config/compose/services/elastic-agent/cloud/docker-compose.yml
	modified:   cli/config/compose/services/elastic-agent/debian/docker-compose.yml
	modified:   cli/config/compose/services/elastic-agent/docker-compose.yml
	modified:   cli/config/compose/services/elasticsearch/docker-compose.yml
	modified:   cli/config/compose/services/fleet-server-centos/docker-compose.yml
	modified:   cli/config/compose/services/fleet-server-debian/docker-compose.yml
	modified:   cli/config/compose/services/metricbeat/docker-compose.yml
	modified:   e2e/_suites/fleet/features/agent_endpoint_integration.feature
	modified:   e2e/_suites/fleet/ingest_manager_test.go
	modified:   e2e/_suites/metricbeat/metricbeat_test.go
	modified:   e2e/commons-test.mk
	new file:   internal/_testresources/beats/x-pack/elastic-agent/build/distributions/elastic-agent-8.0.0-SNAPSHOT-aarch64.rpm
	new file:   internal/_testresources/beats/x-pack/elastic-agent/build/distributions/elastic-agent-8.0.0-SNAPSHOT-arm64.deb
	new file:   internal/_testresources/beats/x-pack/elastic-agent/build/distributions/elastic-agent-8.0.0-SNAPSHOT-linux-arm64.docker.tar.gz
	new file:   internal/_testresources/beats/x-pack/elastic-agent/build/distributions/elastic-agent-8.0.0-SNAPSHOT-linux-arm64.tar.gz
	new file:   internal/_testresources/beats/x-pack/elastic-agent/build/distributions/elastic-agent-8.0.0-SNAPSHOT-linux-x86_64.tar.gz
	new file:   internal/_testresources/beats/x-pack/elastic-agent/build/distributions/elastic-agent-ubi8-8.0.0-SNAPSHOT-linux-arm64.docker.tar.gz
	modified:   internal/deploy/docker.go
	modified:   internal/installer/base_test.go
	modified:   internal/utils/utils.go

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	deleted by us:   .ci/packer_cache.sh
	both modified:   cli/config/compose/profiles/fleet/docker-compose.yml
	both modified:   cli/config/compose/profiles/metricbeat/docker-compose.yml
	both modified:   e2e/_suites/fleet/stand-alone.go
	both modified:   internal/deploy/docker_client.go
	deleted by us:   internal/installer/elasticagent_deb.go
	deleted by us:   internal/installer/elasticagent_docker.go
	deleted by us:   internal/installer/elasticagent_rpm.go
	deleted by us:   internal/installer/elasticagent_tar.go
	both modified:   internal/utils/utils_test.go

To fix up this pull request, you can check it out locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/


Mergify commands and options

More conditions and actions can be found in the documentation.

You can also trigger Mergify actions by commenting on this pull request:

  • @Mergifyio refresh will re-evaluate the rules
  • @Mergifyio rebase will rebase this PR on its base branch
  • @Mergifyio update will merge the base branch into this PR
  • @Mergifyio backport <destination> will backport this PR on <destination> branch

Additionally, on Mergify dashboard you can:

  • look at your merge queues
  • generate the Mergify configuration with the config editor.

Finally, you can contact us on https://mergify.io/

* feat: add support for running the tests in ARM

* fix: arm workers are not immutable yet

* chore: calculate artifact from the GOARCH variable

At this moment we support amd64 and arm64

* chore: run elasticsearch in the right platform

* chore: add log for architecture

* chore: print env variable for GOARCH

* chore: set GOARCH based in the worker

* fix: wrong copy&paste

* chore: install k8s dependencies on ARM

* chore: install godog using GOARCH

* fix: proper values for GO env in Makefile

* chore: add log when requesting a node

* fix: define Groovy variable

* chore: use proper platform for kibana

* chore: simplify platforms selection logic

* fix: use proper CI label for node selector

* chore: remove running containers in clean-docker script

* chore: clean up docker state on mutable workers

* chore: use parallel branch name for filebeat logs

* fix: add forgotten double quotes

* chore: print Docker compose version On ARM

* chore: use aarch64 architecture for RPM packages

* chore: support ARM in tar installer

* fix: proper arch for TAR installer

* chore: log when the artifact is found

* chore: be defensive about no running containers

* chore: add tests for RPM artifact name

* chore: add tests for x86_64 TAR artifact

* chore: remove docker label from CI workers

* chore: separate ARM tests in a second parallel stage

This pretends to avoid using too many workers, adding a second layer for ARM.

* chore: update stage name

* fix: update arm target platform

* chore: use GOARCH in the junit file

* chore: use GOARCH in filebeat logs

* chore: run all stages in parallel

* chore: add platform to log when skipping tests

* fix: use a filebeat version that supports multiplatform

* chore: do not run integrations on ARM

See elastic/beats#25195

* chore: move test resources for ARM

* chore: use staging environment for package-registry

See https://github.com/elastic/package-storage

* fix: remove clean because arm workers are now immutable

* chore: use new multiplatform images

* chore: use new multiplatform base images for fleet-server

* chore: add platform to elastic-agent cloud flavour

* fix: use proper variable for platform

* chore: do not clean up installer's workspace

Because we are using ephemeral workers, we do not want to remove previous files

* Revert "chore: do not clean up installer's workspace"

This reverts commit 9104501.

* chore(ci): skip pull of images from code

* Revert "chore(ci): skip pull of images from code"

This reverts commit 01cc441.

* chore: set platform when pulling images

* chore: update packer cache with our systemd images

* fix: request proper platform image for the agent

* feat: exclude scenarios per platform

* chore: remove tags for endpoint, as they are not used at all

They are also blocking the addition of a parent tag at the feature level, as maximun number of tags is 3

* fix: skip endpoint integration on ARM64

* feat: provide supported platforms at the scenario level on CI

The scenario will be able to override its parent suite's platforms, making
possible to select different platforms at the scenario level.

* fix: use proper length method for groovy arrays

* fix: groovy arrays use size

* fix: override suite platforms per scenario

* chore: update Jenkins label for arm workers

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

* chore: simplify platforms, reusing stack

* chore: explicitly pass GOARCH to CI script

* Revert "chore: update Jenkins label for arm workers"

This reverts commit 9ef81ca.

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
(cherry picked from commit e1a97f3)

# Conflicts:
#	.ci/packer_cache.sh
#	cli/config/compose/profiles/fleet/docker-compose.yml
#	cli/config/compose/profiles/metricbeat/docker-compose.yml
#	e2e/_suites/fleet/stand-alone.go
#	internal/deploy/docker_client.go
#	internal/installer/elasticagent_deb.go
#	internal/installer/elasticagent_docker.go
#	internal/installer/elasticagent_rpm.go
#	internal/installer/elasticagent_tar.go
#	internal/utils/utils_test.go
@mergify mergify bot added the conflicts There is a conflict in the backported pull request label Jun 1, 2021
@elasticmachine
Copy link
Copy Markdown
Contributor

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #1219 opened

  • Start Time: 2021-06-01T14:52:31.919+0000

  • Duration: 5 min 37 sec

  • Commit: f680d22

Test stats 🧪

Test Results
Failed 0
Passed 11
Skipped 0
Total 11

Trends 🧪

Image of Build Times

Image of Tests

Steps errors 4

Expand to view the steps failures

Install dependencies
  • Took 0 min 16 sec . View more details on here
  • Description: .ci/scripts/install-dependencies.sh
Install dependencies
  • Took 0 min 2 sec . View more details on here
  • Description: .ci/scripts/install-dependencies.sh
Build and test
  • Took 0 min 18 sec . View more details on here
  • Description: .ci/scripts/build-test.sh
Archive the artifacts
  • Took 0 min 0 sec . View more details on here
  • Description: [2021-06-01T14:55:12.582Z] Archiving artifacts script returned exit code 2

Log output

Expand to view the last 100 lines of log output

[2021-06-01T14:56:43.328Z] gopkg.in/src-d/go-git.v4/plumbing/protocol/packp/capability
[2021-06-01T14:56:43.596Z] github.com/emirpasic/gods/trees/binaryheap
[2021-06-01T14:56:43.596Z] gopkg.in/src-d/go-git.v4/storage
[2021-06-01T14:56:43.596Z] gopkg.in/src-d/go-git.v4/plumbing/protocol/packp/sideband
[2021-06-01T14:56:43.596Z] gopkg.in/src-d/go-git.v4/plumbing/format/packfile
[2021-06-01T14:56:43.596Z] gopkg.in/src-d/go-git.v4/utils/merkletrie
[2021-06-01T14:56:43.596Z] github.com/kevinburke/ssh_config
[2021-06-01T14:56:43.596Z] golang.org/x/crypto/ed25519
[2021-06-01T14:56:43.596Z] gopkg.in/src-d/go-git.v4/storage/memory
[2021-06-01T14:56:43.596Z] golang.org/x/crypto/internal/subtle
[2021-06-01T14:56:43.596Z] golang.org/x/crypto/curve25519
[2021-06-01T14:56:43.596Z] golang.org/x/crypto/poly1305
[2021-06-01T14:56:43.596Z] gopkg.in/src-d/go-git.v4/plumbing/object
[2021-06-01T14:56:43.596Z] golang.org/x/crypto/chacha20
[2021-06-01T14:56:43.596Z] gopkg.in/src-d/go-git.v4/plumbing/protocol/packp
[2021-06-01T14:56:43.856Z] golang.org/x/crypto/blowfish
[2021-06-01T14:56:43.856Z] gopkg.in/src-d/go-git.v4/utils/merkletrie/filesystem
[2021-06-01T14:56:43.856Z] gopkg.in/src-d/go-git.v4/utils/merkletrie/index
[2021-06-01T14:56:43.856Z] github.com/elastic/e2e-testing/internal/sanitizer
[2021-06-01T14:56:43.856Z] gopkg.in/src-d/go-git.v4/plumbing/format/objfile
[2021-06-01T14:56:43.856Z] golang.org/x/crypto/ssh/internal/bcrypt_pbkdf
[2021-06-01T14:56:43.856Z] github.com/spf13/pflag
[2021-06-01T14:56:43.856Z] golang.org/x/crypto/ssh
[2021-06-01T14:56:43.856Z] gopkg.in/src-d/go-git.v4/storage/filesystem/dotgit
[2021-06-01T14:56:44.143Z] gopkg.in/src-d/go-git.v4/plumbing/transport
[2021-06-01T14:56:44.143Z] gopkg.in/src-d/go-git.v4/plumbing/revlist
[2021-06-01T14:56:44.143Z] gopkg.in/src-d/go-git.v4/plumbing/transport/internal/common
[2021-06-01T14:56:44.143Z] gopkg.in/src-d/go-git.v4/storage/filesystem
[2021-06-01T14:56:44.408Z] gopkg.in/src-d/go-git.v4/plumbing/transport/git
[2021-06-01T14:56:44.408Z] gopkg.in/src-d/go-git.v4/plumbing/transport/http
[2021-06-01T14:56:44.669Z] gopkg.in/src-d/go-git.v4/plumbing/transport/server
[2021-06-01T14:56:44.669Z] gopkg.in/src-d/go-git.v4/plumbing/transport/file
[2021-06-01T14:56:44.669Z] github.com/spf13/cobra
[2021-06-01T14:56:44.928Z] google.golang.org/protobuf/internal/filetype
[2021-06-01T14:56:44.928Z] google.golang.org/protobuf/runtime/protoimpl
[2021-06-01T14:56:44.928Z] golang.org/x/crypto/ssh/knownhosts
[2021-06-01T14:56:44.928Z] golang.org/x/crypto/ssh/agent
[2021-06-01T14:56:44.928Z] google.golang.org/protobuf/types/known/anypb
[2021-06-01T14:56:44.928Z] google.golang.org/protobuf/types/known/durationpb
[2021-06-01T14:56:44.928Z] google.golang.org/protobuf/types/known/timestamppb
[2021-06-01T14:56:44.928Z] github.com/golang/protobuf/proto
[2021-06-01T14:56:44.928Z] github.com/golang/protobuf/ptypes/any
[2021-06-01T14:56:44.929Z] github.com/golang/protobuf/ptypes/timestamp
[2021-06-01T14:56:44.929Z] github.com/golang/protobuf/ptypes/duration
[2021-06-01T14:56:45.189Z] github.com/xanzy/ssh-agent
[2021-06-01T14:56:45.189Z] gopkg.in/src-d/go-git.v4/plumbing/transport/ssh
[2021-06-01T14:56:45.189Z] gopkg.in/src-d/go-git.v4/plumbing/transport/client
[2021-06-01T14:56:45.189Z] google.golang.org/genproto/googleapis/rpc/status
[2021-06-01T14:56:45.189Z] github.com/golang/protobuf/ptypes
[2021-06-01T14:56:45.189Z] gopkg.in/src-d/go-git.v4
[2021-06-01T14:56:45.189Z] google.golang.org/grpc/internal/status
[2021-06-01T14:56:45.189Z] google.golang.org/grpc/status
[2021-06-01T14:56:45.450Z] github.com/containerd/containerd/errdefs
[2021-06-01T14:56:45.450Z] github.com/containerd/containerd/platforms
[2021-06-01T14:56:45.450Z] github.com/docker/docker/errdefs
[2021-06-01T14:56:45.450Z] github.com/docker/docker/client
[2021-06-01T14:56:45.711Z] github.com/elastic/e2e-testing/internal/git
[2021-06-01T14:56:45.971Z] github.com/testcontainers/testcontainers-go
[2021-06-01T14:56:45.971Z] github.com/elastic/e2e-testing/internal/deploy
[2021-06-01T14:56:45.971Z] # github.com/elastic/e2e-testing/internal/deploy
[2021-06-01T14:56:45.971Z] ../internal/deploy/docker_client.go:464:1: syntax error: unexpected <<, expecting }
[2021-06-01T14:56:45.971Z] ../internal/deploy/docker_client.go:476:66: invalid character U+0023 '#'
[2021-06-01T14:56:45.971Z] Makefile:11: recipe for target 'install' failed
[2021-06-01T14:56:45.971Z] make: *** [install] Error 2
[2021-06-01T14:56:45.971Z] make: Leaving directory '/var/lib/jenkins/workspace/2e-tests_e2e-testing-mbp_PR-1219/src/github.com/elastic/e2e-testing/cli'
[2021-06-01T14:56:46.007Z] Sleeping for 5 sec
[2021-06-01T14:56:51.020Z] ERROR: script returned exit code 2
[2021-06-01T14:56:51.020Z] Retrying
[2021-06-01T14:56:51.381Z] + .ci/scripts/install-dependencies.sh
[2021-06-01T14:56:51.381Z] + curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh
[2021-06-01T14:56:51.381Z] ++ go env GOPATH
[2021-06-01T14:56:51.381Z] + sh -s -- -b /var/lib/jenkins/workspace/2e-tests_e2e-testing-mbp_PR-1219/bin v1.34.1
[2021-06-01T14:56:51.381Z] golangci/golangci-lint info checking GitHub for tag 'v1.34.1'
[2021-06-01T14:56:51.641Z] golangci/golangci-lint info found version: 1.34.1 for v1.34.1/linux/amd64
[2021-06-01T14:56:52.580Z] golangci/golangci-lint info installed /var/lib/jenkins/workspace/2e-tests_e2e-testing-mbp_PR-1219/bin/golangci-lint
[2021-06-01T14:56:52.580Z] + go get -v golang.org/x/lint/golint
[2021-06-01T14:56:52.580Z] go: found golang.org/x/lint/golint in golang.org/x/lint v0.0.0-20210508222113-6edffad5e616
[2021-06-01T14:56:52.840Z] + make -C cli install
[2021-06-01T14:56:52.840Z] make: Entering directory '/var/lib/jenkins/workspace/2e-tests_e2e-testing-mbp_PR-1219/src/github.com/elastic/e2e-testing/cli'
[2021-06-01T14:56:52.840Z] go get -v -t ./...
[2021-06-01T14:56:53.409Z] github.com/elastic/e2e-testing/internal/deploy
[2021-06-01T14:56:53.409Z] # github.com/elastic/e2e-testing/internal/deploy
[2021-06-01T14:56:53.409Z] ../internal/deploy/docker_client.go:464:1: syntax error: unexpected <<, expecting }
[2021-06-01T14:56:53.409Z] ../internal/deploy/docker_client.go:476:66: invalid character U+0023 '#'
[2021-06-01T14:56:53.409Z] Makefile:11: recipe for target 'install' failed
[2021-06-01T14:56:53.409Z] make: *** [install] Error 2
[2021-06-01T14:56:53.409Z] make: Leaving directory '/var/lib/jenkins/workspace/2e-tests_e2e-testing-mbp_PR-1219/src/github.com/elastic/e2e-testing/cli'
[2021-06-01T14:56:53.443Z] Sleeping for 10 sec
[2021-06-01T14:57:05.314Z] Failed in branch Sanity checks
[2021-06-01T14:57:05.539Z] Stage "Build Docs" skipped due to earlier failure(s)
[2021-06-01T14:57:05.677Z] Stage "End-To-End Tests" skipped due to earlier failure(s)
[2021-06-01T14:57:05.841Z] Stage "Release" skipped due to earlier failure(s)
[2021-06-01T14:57:07.794Z] Running on worker-395930 in /var/lib/jenkins/workspace/2e-tests_e2e-testing-mbp_PR-1219
[2021-06-01T14:57:07.981Z] [INFO] getVaultSecret: Getting secrets
[2021-06-01T14:57:08.053Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-06-01T14:57:10.532Z] + chmod 755 generate-build-data.sh
[2021-06-01T14:57:10.532Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/e2e-tests/e2e-testing-mbp/PR-1219/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/e2e-tests/e2e-testing-mbp/PR-1219/runs/1 FAILURE 277203
[2021-06-01T14:57:10.532Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/e2e-tests/e2e-testing-mbp/PR-1219/runs/1/steps/?limit=10000 -o steps-info.json
[2021-06-01T14:57:12.794Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/e2e-tests/e2e-testing-mbp/PR-1219/runs/1/tests/?status=FAILED -o tests-errors.json
[2021-06-01T14:57:12.794Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/e2e-tests/e2e-testing-mbp/PR-1219/runs/1/log/ -o pipeline-log.txt

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 11
Skipped 0
Total 11

@mdelapenya
Copy link
Copy Markdown
Contributor

Waiting for #1212 to be merged first

@mergify
Copy link
Copy Markdown
Contributor Author

mergify bot commented Jun 1, 2021

This pull request is now in conflict. Could you fix it @mergify[bot]? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b mergify/bp/7.x/pr-707 upstream/mergify/bp/7.x/pr-707
git merge upstream/7.x
git push upstream mergify/bp/7.x/pr-707

@mdelapenya
Copy link
Copy Markdown
Contributor

@Mergifyio refresh

@mergify
Copy link
Copy Markdown
Contributor Author

mergify bot commented Jun 1, 2021

Command refresh: success

Pull request refreshed

@mdelapenya mdelapenya closed this Jun 1, 2021
@mergify mergify bot deleted the mergify/bp/7.x/pr-707 branch June 1, 2021 15:34
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

conflicts There is a conflict in the backported pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants