chore: define provider for each test suite by mdelapenya · Pull Request #2213 · elastic/e2e-testing

mdelapenya · 2022-03-09T08:23:40Z

chore: preset the provider at suite or/and scenario level
docs: document the CI descriptor for the platform support
docs: describe each platform
fix: typo
chore: declare platforms just once
docs: document machine dependencies

What does this PR do?

This PR does three things:

adds support for a test suite and/or a test scenario to declare the provider to be used in the test execution on CI
splits the descriptor files to have one single file for the available platforms at CI time.
documents the file descriptors so that anybody can understand how to add a new supported platform

Why is it important?

First, document the descriptors. This is key for external contributors owning their own support matrix.

Second, prepare the code to re-enable skipped suites using docker or elastic-package providers

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have run the Unit tests (make unit-test), and they are passing locally
I have run the End-2-End tests for the suite I'm working on, and they are passing locally
I have noticed new Go dependencies (run make notice in the proper directory)

Related issues

Will enable Use elastic-package again #1737 Re-enable "Deploying the "<os>" agent" scenario #1658
Closes Document the e2e-tests.yaml file #1657

elasticmachine · 2022-03-09T08:28:53Z

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
[](https://ci-stats.elastic.co/app/apm/services/beats-ci/transactions/view?rangeFrom=2022-03-09T17:20:12.562Z&rangeTo=2022-03-09T17:40:12.562Z&transactionName=BUILD e2e-tests/e2e-testing-mbp/PR-{number}&transactionType=job&latencyAggregationType=avg&traceId=f27a5c14f4436a9da198538a765f0364&transactionId=4879c503d7a51443)

Expand to view the summary

Build stats

Start Time: 2022-03-09T17:30:12.562+0000
Duration: 50 min 59 sec

Test stats 🧪

Test	Results
Failed	1
Passed	275
Skipped	0
Total	276

Test errors

Expand to view the tests failures

`Initializing / End-To-End Tests / kubernetes-autodiscover_debian_amd64_elastic-agent / Logs collection from running pod – elastic-agent standalone`

Expand to view the error details

 Step "elastic-agent" collects events with "kubernetes.pod.name:a-pod": context deadline exceeded

no stacktrace

Steps errors

Expand to view the steps failures

`Shell Script`

Took 16 min 20 sec . View more details here
Description: ssh -tt -o TCPKeepAlive=yes -o ServerAliveInterval=60 -o ServerAliveCountMax=10 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i /var/lib/jenkins/workspace/PR-2213-7-cf3141e0-6ca7-4886-885e-7437b78ce39b/e2essh admin@18.218.203.128 -- 'sudo bash /home/admin/e2e-testing/.ci/scripts/functional-test.sh "elastic-agent && ~@nightly && ~@skip:amd64" '

`Archive the artifacts`

Took 0 min 1 sec . View more details here
Description: [2022-03-09T18:17:44.140Z] Archiving artifacts script returned exit code 2

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Genuine test errors

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

Name: Initializing / End-To-End Tests / kubernetes-autodiscover_debian_amd64_elastic-agent / Logs collection from running pod – elastic-agent standalone

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

adam-stokes

Very cool!!

mdelapenya · 2022-03-09T14:31:55Z

.ci/.e2e-platforms.yaml

+    image: "ami-0f7cb53c916a75006"
+    instance_type: "c5.4xlarge"
+    username: ec2-user
+  fleet_elastic_pkg:


I think we can remove this platform in a follow-up PR, as we can control the same with provider: elastic-package in the YAML file

This reverts commit cbb281f.

mdelapenya · 2022-03-09T17:30:47Z

.ci/.e2e-tests-for-elastic-agent.yaml

-      # - name: "Stand-alone"
-      #   tags: "stand_alone_agent"
-      #   platforms: ["fleet_elastic_pkg"]
+      - name: "Stand-alone"


I'm going to revert this commit and add it in a follow up, as the stand-alone tests are failing, even locally

I think we are messing how the elastic-agent Docker container is bootstrapped when running with Docker provider cc/ @adam-stokes

* chore: preset the provider at suite or/and scenario level * docs: document the CI descriptor for the platform support * docs: describe each platform * fix: typo * chore: declare platforms just once * docs: document machine dependencies * docs: link to CI infra documentation * fix: use right location for the platforms file * fix: set remote variables only for remote provider * fix: disable fleet-server for regular standalone deployments * chore: add ES and kibana endpoints to the logs * chore: apply docker provider for elastic-agent triggers * chore: re-enable stand-alone scenarios * chore: move fleet-server default policy to a constant * Revert "chore: re-enable stand-alone scenarios" This reverts commit cbb281f. (cherry picked from commit 2ddece6) # Conflicts: # .ci/.e2e-tests-for-elastic-agent.yaml # .ci/.e2e-tests.yaml

* chore: preset the provider at suite or/and scenario level * docs: document the CI descriptor for the platform support * docs: describe each platform * fix: typo * chore: declare platforms just once * docs: document machine dependencies * docs: link to CI infra documentation * fix: use right location for the platforms file * fix: set remote variables only for remote provider * fix: disable fleet-server for regular standalone deployments * chore: add ES and kibana endpoints to the logs * chore: apply docker provider for elastic-agent triggers * chore: re-enable stand-alone scenarios * chore: move fleet-server default policy to a constant * Revert "chore: re-enable stand-alone scenarios" This reverts commit cbb281f. (cherry picked from commit 2ddece6) # Conflicts: # .ci/.e2e-tests-for-elastic-agent.yaml # .ci/.e2e-tests.yaml # e2e/_suites/fleet/fleet.go

* chore: preset the provider at suite or/and scenario level * docs: document the CI descriptor for the platform support * docs: describe each platform * fix: typo * chore: declare platforms just once * docs: document machine dependencies * docs: link to CI infra documentation * fix: use right location for the platforms file * fix: set remote variables only for remote provider * fix: disable fleet-server for regular standalone deployments * chore: add ES and kibana endpoints to the logs * chore: apply docker provider for elastic-agent triggers * chore: re-enable stand-alone scenarios * chore: move fleet-server default policy to a constant * Revert "chore: re-enable stand-alone scenarios" This reverts commit cbb281f. (cherry picked from commit 2ddece6) # Conflicts: # .ci/.e2e-tests-for-elastic-agent.yaml # .ci/.e2e-tests.yaml # README.md # e2e/_suites/fleet/fleet.go

) * chore: define provider for each test suite (#2213) * chore: preset the provider at suite or/and scenario level * docs: document the CI descriptor for the platform support * docs: describe each platform * fix: typo * chore: declare platforms just once * docs: document machine dependencies * docs: link to CI infra documentation * fix: use right location for the platforms file * fix: set remote variables only for remote provider * fix: disable fleet-server for regular standalone deployments * chore: add ES and kibana endpoints to the logs * chore: apply docker provider for elastic-agent triggers * chore: re-enable stand-alone scenarios * chore: move fleet-server default policy to a constant * Revert "chore: re-enable stand-alone scenarios" This reverts commit cbb281f. (cherry picked from commit 2ddece6) # Conflicts: # .ci/.e2e-tests-for-elastic-agent.yaml # .ci/.e2e-tests.yaml # README.md # e2e/_suites/fleet/fleet.go * fix: resolve conflicts Co-authored-by: Manuel de la Peña <mdelapenya@gmail.com>

* chore: define provider for each test suite (#2213) * chore: preset the provider at suite or/and scenario level * docs: document the CI descriptor for the platform support * docs: describe each platform * fix: typo * chore: declare platforms just once * docs: document machine dependencies * docs: link to CI infra documentation * fix: use right location for the platforms file * fix: set remote variables only for remote provider * fix: disable fleet-server for regular standalone deployments * chore: add ES and kibana endpoints to the logs * chore: apply docker provider for elastic-agent triggers * chore: re-enable stand-alone scenarios * chore: move fleet-server default policy to a constant * Revert "chore: re-enable stand-alone scenarios" This reverts commit cbb281f. (cherry picked from commit 2ddece6) # Conflicts: # .ci/.e2e-tests-for-elastic-agent.yaml # .ci/.e2e-tests.yaml # e2e/_suites/fleet/fleet.go * fix: resolve conflicts Co-authored-by: Manuel de la Peña <mdelapenya@gmail.com>

* chore: define provider for each test suite (#2213) * chore: preset the provider at suite or/and scenario level * docs: document the CI descriptor for the platform support * docs: describe each platform * fix: typo * chore: declare platforms just once * docs: document machine dependencies * docs: link to CI infra documentation * fix: use right location for the platforms file * fix: set remote variables only for remote provider * fix: disable fleet-server for regular standalone deployments * chore: add ES and kibana endpoints to the logs * chore: apply docker provider for elastic-agent triggers * chore: re-enable stand-alone scenarios * chore: move fleet-server default policy to a constant * Revert "chore: re-enable stand-alone scenarios" This reverts commit cbb281f. (cherry picked from commit 2ddece6) # Conflicts: # .ci/.e2e-tests-for-elastic-agent.yaml # .ci/.e2e-tests.yaml * fix: resolve conflicts Co-authored-by: Manuel de la Peña <mdelapenya@gmail.com>

* main: (268 commits) bump stack version 8.2.0-ff67d7b8 (#2242) ci: periodic builds for the last two minor versions (#2241) bump stack version 8.2.0-9bac538c (#2240) fix: add end-of-file line in templates (#2237) chore: add Christos to SSH users bump stack version 8.2.0-fee3b8d2 (#2234) Separate ES workload from Agent spec and make it req for the scenarios (#2203) bump stack version 8.2.0-d02c907a (#2228) bump stack version 8.2.0-63265ec9 (#2225) fix: wrong link (#2220) chore: define provider for each test suite (#2213) ci: provision e2-small workers (#2212) bump stack version 8.2.0-a12f7069 (#2209) chore: support for Oracle Linux 8 (#2208) bump stack version 8.2.0-190af159 (#2206) bump stack version 8.2.0-fdde08ec (#2200) bump stack version 8.2.0-43df679f (#2193) chore: use trace level in logs (#2130) bump stack version 8.2.0-bdf2ad74 (#2188) feat: support downloading project artifacts for the new bucket layout (#2172) ...

mdelapenya added 6 commits March 9, 2022 08:20

chore: preset the provider at suite or/and scenario level

b657fdc

docs: document the CI descriptor for the platform support

0cc7256

docs: describe each platform

57d717d

fix: typo

03577a8

chore: declare platforms just once

bbc6f22

docs: document machine dependencies

77a2e0d

mdelapenya added backport-v7.17.0 Automated backport with mergify backport-v8.0.0 Automated backport with mergify backport-v8.1.0 Automated backport with mergify labels Mar 9, 2022

mdelapenya self-assigned this Mar 9, 2022

mdelapenya requested a review from a team March 9, 2022 08:23

mdelapenya added 5 commits March 9, 2022 09:31

docs: link to CI infra documentation

7615c37

fix: use right location for the platforms file

e6408b2

fix: set remote variables only for remote provider

7ac0811

fix: disable fleet-server for regular standalone deployments

e459361

chore: add ES and kibana endpoints to the logs

2a81f0b

mdelapenya marked this pull request as ready for review March 9, 2022 13:10

Merge branch 'main' into adjust-provisioners-docker-remote-provider

eaf7825

adam-stokes approved these changes Mar 9, 2022

View reviewed changes

mdelapenya added 2 commits March 9, 2022 15:29

chore: apply docker provider for elastic-agent triggers

0db4edb

chore: re-enable stand-alone scenarios

cbb281f

mdelapenya commented Mar 9, 2022

View reviewed changes

mdelapenya added 2 commits March 9, 2022 17:22

chore: move fleet-server default policy to a constant

64ae498

Revert "chore: re-enable stand-alone scenarios"

38d1ea8

This reverts commit cbb281f.

mdelapenya commented Mar 9, 2022

View reviewed changes

mdelapenya merged commit 2ddece6 into elastic:main Mar 9, 2022

mdelapenya deleted the adjust-provisioners-docker-remote-provider branch March 9, 2022 18:22

mergify bot mentioned this pull request Mar 9, 2022

[8.1](backport #2213) chore: define provider for each test suite #2217

Merged

mergify bot mentioned this pull request Mar 9, 2022

[8.0](backport #2213) chore: define provider for each test suite #2218

Merged

mergify bot mentioned this pull request Mar 9, 2022

[7.17](backport #2213) chore: define provider for each test suite #2219

Merged

mdelapenya mentioned this pull request Mar 10, 2022

fix: wrong link #2220

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: define provider for each test suite#2213

chore: define provider for each test suite#2213
mdelapenya merged 16 commits intoelastic:mainfrom
mdelapenya:adjust-provisioners-docker-remote-provider

mdelapenya commented Mar 9, 2022

Uh oh!

elasticmachine commented Mar 9, 2022 •

edited

Loading

Build stats

Test stats 🧪

`Initializing / End-To-End Tests / kubernetes-autodiscover_debian_amd64_elastic-agent / Logs collection from running pod – elastic-agent standalone`

`Shell Script`

`Archive the artifacts`

Genuine test errors

Uh oh!

adam-stokes left a comment

Uh oh!

mdelapenya Mar 9, 2022

Uh oh!

mdelapenya Mar 9, 2022 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

mdelapenya commented Mar 9, 2022

What does this PR do?

Why is it important?

Checklist

Related issues

Uh oh!

elasticmachine commented Mar 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Tests Failed

Build stats

Test stats 🧪

Test errors

Initializing / End-To-End Tests / kubernetes-autodiscover_debian_amd64_elastic-agent / Logs collection from running pod – elastic-agent standalone

Steps errors

Shell Script

Archive the artifacts

🐛 Flaky test report

Genuine test errors

🤖 GitHub comments

Uh oh!

adam-stokes left a comment

Choose a reason for hiding this comment

Uh oh!

mdelapenya Mar 9, 2022

Choose a reason for hiding this comment

Uh oh!

mdelapenya Mar 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

elasticmachine commented Mar 9, 2022 •

edited

Loading

`Initializing / End-To-End Tests / kubernetes-autodiscover_debian_amd64_elastic-agent / Logs collection from running pod – elastic-agent standalone`

`Shell Script`

`Archive the artifacts`

mdelapenya Mar 9, 2022 •

edited

Loading