feat: support downloading project artifacts for the new bucket layout by mdelapenya · Pull Request #2172 · elastic/e2e-testing

mdelapenya · 2022-02-23T16:23:58Z

What does this PR do?

It exposes a new method: FetchProjectBinary so that the existing FetchBeatsBinary calls the new one with the required arguments.

This method will use the new bucket layout:

$project
- commits
  - SHA1
  - binary
- snapshots
  - binary-version

And it will look up the artifacts in the three possible locations:

the new project layout (as shown above) in the fleet-ci bucket
the new project layout (as shown above) in the beats-ci bucket
the new Beats layout (same as legacy but adding a beats parent dir)
the legacy Beats layout

This PR is decoupling the bucket resolution into the concept of URL Resolvers, so that the logic is narrowed to the scope of that struct. We are building three resolvers, one for each layout. We have refactored the existing code that calculated the bucket URL using the new resolvers abstraction, checking that the existing unit tests covered the changes. Finally, we moved those tests to the right Go file, so that it's easier to maintain them.

It's important to mention that we did not change the API contracts in the pkg package: we are aware of they are consumed (by elastic-agent i.e.) and we are keeping old functionality using defaults. The framework is using new methods that support the new expected behaviors:

selectively use the CI snapshots from the bucket for downloads: when installing elastic-agent on top of beats, we need to install metricbeat and filebeat first. Their versions are going to be aligned with the stack, so we need to fetch them from the Elastic artifactory.
fetch CI artifacts for a project, in an exhaustive manner: we will try all possible locations to fetch the CI snapshots, being possible to use the Fleet CI bucket or the Beats CI one, in the new and legacy layout.

Apart from that, we realised that the VM support did not migrate the ability to consume a Git SHA from Beats. In this PR we are adding back that support, simply passing the env vars defined at the Jenkinsfile to the Ansible playbook to generate the right provisioning script.

Why is it important?

Both fleet-server and elastic-agent projects will follow this structure, and because we want to consume the artifacts for PRs, then we need to update the method to look up the bucket.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have run the Unit tests (make unit-test), and they are passing locally
I have run the End-2-End tests for the suite I'm working on, and they are passing locally
I have noticed new Go dependencies (run make notice in the proper directory)

Related issues

Closes Support downloading artifacts from a project-based structure in the GCP bucket #2149

Other concerns

We are aware that passing the GIT_SHA of the elastic-agent could cause errors in the k8s-autodiscover test suite, because they download the binaries for metricbeat and filebeat in that commit. Because the SHA for the elastic-agent comes from separate repository, the binaries for MB and FB don't exist for that commit. In #2175 we explain it more in depth

Old PRs needs to be tested and the binaries could not exist in the new layout

1. new project layout 2. new beats layout 3. legacy beats layout

narph

LGTM

elasticmachine · 2022-02-23T16:32:55Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-02-28T17:01:37.712+0000
Duration: 42 min 9 sec

Test stats 🧪

Test	Results
Failed	0
Passed	101
Skipped	0
Total	101

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

adam-stokes · 2022-02-23T17:28:53Z

pkg/downloads/versions.go

+		// look up the bucket in this particular order:
+		// 1. the project layout (elastic-agent, fleet-server)
+		// 2. the new beats layout (beats)
+		// 3. the legacy Beats layout (commits/snapshots)


adam-stokes

Looks awesome! 🥳

mdelapenya · 2022-02-23T18:55:42Z

Need to run it for a commit Sha, will try later on

.ci/ansible/tasks/setup_test_script.yml

mdelapenya · 2022-02-24T21:04:15Z

@adam-stokes I think this is ready to merge. Please do a final review 👀 with latest changes 🙏

The 13 errors are related to k8s-autodiscover for a SHA commit: because the last build used a SHA on elastic-agent repo, the binaries used for the k8s feature are not found by any resolver, as the SHA commit does not exist in the Beats repository (and its bucket). In #2175 I explain the need to refactor the layout of the test suites.

mdelapenya · 2022-02-28T16:59:48Z

Tests for a commit on Beats

Commit: 37d07d1cd3ea761939fbe5f568b747511b302c0c
CI Build: https://beats-ci.elastic.co/blue/organizations/jenkins/e2e-tests%2Fe2e-testing-mbp/detail/PR-2172/27/pipeline
Bucket: beats-ci-artifacts/commits/37d07d1cd3ea761939fbe5f568b747511b302c0c
Should use elastic-agent in STACK_VERSION

Tests for a commit on Elastic Agent

Commit: e921876ddd33cf6053b1487d7b32e17965b79159
CI Build: https://beats-ci.elastic.co/blue/organizations/jenkins/e2e-tests%2Fe2e-testing-mbp/detail/PR-2172/28/pipeline
Bucket: fleet-ci-artifacts/elastic-agent/commits/e921876ddd33cf6053b1487d7b32e17965b79159
Should use beats (filebeat, metricbeat) in STACK_VERSION

mdelapenya · 2022-02-28T18:07:10Z

pkg/downloads/buckets.go

+	if strings.EqualFold(r.Variant, "ubi8") {
+		artifact = strings.ReplaceAll(artifact, "-ubi8", "")
+	}


Candidate to be extracted, as it's used across all resolvers

mdelapenya · 2022-02-28T18:07:17Z

pkg/downloads/buckets.go

+	ciSnapshotsFn := UseBeatsCISnapshots
+	if strings.EqualFold(artifact, "elastic-agent") {
+		ciSnapshotsFn = UseElasticAgentCISnapshots
+	}


Candidate to be extracted, as it's used across all resolvers

…#2172) * feat: support for the new bucket structure * feat: calculate bucket URLs using resolvers * chore: support for legacy Beats Old PRs needs to be tested and the binaries could not exist in the new layout * chore: support looking up the bucket in up-to 3 locations 1. new project layout 2. new beats layout 3. legacy beats layout * chore: simplify boolean logic * fix: pass worker environment for versions to the VMs * fix: predefine kibana version * fix: remove extra char * fix: remove extra char * chore: log resolvers * chore: use new project method when fetching artifacts * fix: pass stack version to stack deployment * fix: use artifact to check for ubi8 variant * chore: improve logs in resolvers * fix: apply variants to project resolver Ubi8 is a valid variant for elastic-agent * chore: use Stack version for the dependant beats In the running-on-top-of-beats feature file, the code installs some Beats and we do not want to use the given SHA, because elastic-agent now lives in a separate repo, therefore they do not share commit SHAs * fix: use fleet-ci bucket for elastic-agent and fleet-server * chore: look up both CI buckets * chore: make useCISnapshots configurable on downloads * chore: background processes should not use CI snapshots * feat: separate elastic-agent version from beats version * chore: extract useCISnapshots to a function * chore: use Github_Check_Repo to identify where the Git commit lives * fix: k8s-autodiscover lives in both worlds * fix: missing package * fix: use snapshot version in k8s-autodiscover (cherry picked from commit 044dedf) # Conflicts: # .ci/Jenkinsfile

…#2172) * feat: support for the new bucket structure * feat: calculate bucket URLs using resolvers * chore: support for legacy Beats Old PRs needs to be tested and the binaries could not exist in the new layout * chore: support looking up the bucket in up-to 3 locations 1. new project layout 2. new beats layout 3. legacy beats layout * chore: simplify boolean logic * fix: pass worker environment for versions to the VMs * fix: predefine kibana version * fix: remove extra char * fix: remove extra char * chore: log resolvers * chore: use new project method when fetching artifacts * fix: pass stack version to stack deployment * fix: use artifact to check for ubi8 variant * chore: improve logs in resolvers * fix: apply variants to project resolver Ubi8 is a valid variant for elastic-agent * chore: use Stack version for the dependant beats In the running-on-top-of-beats feature file, the code installs some Beats and we do not want to use the given SHA, because elastic-agent now lives in a separate repo, therefore they do not share commit SHAs * fix: use fleet-ci bucket for elastic-agent and fleet-server * chore: look up both CI buckets * chore: make useCISnapshots configurable on downloads * chore: background processes should not use CI snapshots * feat: separate elastic-agent version from beats version * chore: extract useCISnapshots to a function * chore: use Github_Check_Repo to identify where the Git commit lives * fix: k8s-autodiscover lives in both worlds * fix: missing package * fix: use snapshot version in k8s-autodiscover (cherry picked from commit 044dedf) # Conflicts: # .ci/Jenkinsfile # e2e/_suites/fleet/fleet.go # e2e/_suites/fleet/stand-alone.go # e2e/_suites/kubernetes-autodiscover/autodiscover_test.go # internal/buckets.go # internal/buckets_test.go # internal/deploy/docker.go # internal/installer/elasticagent_deb.go # internal/installer/elasticagent_docker.go # internal/installer/elasticagent_rpm.go # internal/installer/elasticagent_tar.go # internal/installer/elasticagent_tar_macos.go # internal/installer/elasticagent_zip.go # internal/versions.go

… the new bucket layout (#2182) * feat: support downloading project artifacts for the new bucket layout (#2172) * feat: support for the new bucket structure * feat: calculate bucket URLs using resolvers * chore: support for legacy Beats Old PRs needs to be tested and the binaries could not exist in the new layout * chore: support looking up the bucket in up-to 3 locations 1. new project layout 2. new beats layout 3. legacy beats layout * chore: simplify boolean logic * fix: pass worker environment for versions to the VMs * fix: predefine kibana version * fix: remove extra char * fix: remove extra char * chore: log resolvers * chore: use new project method when fetching artifacts * fix: pass stack version to stack deployment * fix: use artifact to check for ubi8 variant * chore: improve logs in resolvers * fix: apply variants to project resolver Ubi8 is a valid variant for elastic-agent * chore: use Stack version for the dependant beats In the running-on-top-of-beats feature file, the code installs some Beats and we do not want to use the given SHA, because elastic-agent now lives in a separate repo, therefore they do not share commit SHAs * fix: use fleet-ci bucket for elastic-agent and fleet-server * chore: look up both CI buckets * chore: make useCISnapshots configurable on downloads * chore: background processes should not use CI snapshots * feat: separate elastic-agent version from beats version * chore: extract useCISnapshots to a function * chore: use Github_Check_Repo to identify where the Git commit lives * fix: k8s-autodiscover lives in both worlds * fix: missing package * fix: use snapshot version in k8s-autodiscover (cherry picked from commit 044dedf) # Conflicts: # .ci/Jenkinsfile * fix: resolve conflicts Co-authored-by: Manuel de la Peña <mdelapenya@gmail.com>

… the new bucket layout (#2181) * feat: support downloading project artifacts for the new bucket layout (#2172) * feat: support for the new bucket structure * feat: calculate bucket URLs using resolvers * chore: support for legacy Beats Old PRs needs to be tested and the binaries could not exist in the new layout * chore: support looking up the bucket in up-to 3 locations 1. new project layout 2. new beats layout 3. legacy beats layout * chore: simplify boolean logic * fix: pass worker environment for versions to the VMs * fix: predefine kibana version * fix: remove extra char * fix: remove extra char * chore: log resolvers * chore: use new project method when fetching artifacts * fix: pass stack version to stack deployment * fix: use artifact to check for ubi8 variant * chore: improve logs in resolvers * fix: apply variants to project resolver Ubi8 is a valid variant for elastic-agent * chore: use Stack version for the dependant beats In the running-on-top-of-beats feature file, the code installs some Beats and we do not want to use the given SHA, because elastic-agent now lives in a separate repo, therefore they do not share commit SHAs * fix: use fleet-ci bucket for elastic-agent and fleet-server * chore: look up both CI buckets * chore: make useCISnapshots configurable on downloads * chore: background processes should not use CI snapshots * feat: separate elastic-agent version from beats version * chore: extract useCISnapshots to a function * chore: use Github_Check_Repo to identify where the Git commit lives * fix: k8s-autodiscover lives in both worlds * fix: missing package * fix: use snapshot version in k8s-autodiscover (cherry picked from commit 044dedf) # Conflicts: # .ci/Jenkinsfile * fix: resolve conflicts Co-authored-by: Manuel de la Peña <mdelapenya@gmail.com>

…elastic#2172) * feat: support for the new bucket structure * feat: calculate bucket URLs using resolvers * chore: support for legacy Beats Old PRs needs to be tested and the binaries could not exist in the new layout * chore: support looking up the bucket in up-to 3 locations 1. new project layout 2. new beats layout 3. legacy beats layout * chore: simplify boolean logic * fix: pass worker environment for versions to the VMs * fix: predefine kibana version * fix: remove extra char * fix: remove extra char * chore: log resolvers * chore: use new project method when fetching artifacts * fix: pass stack version to stack deployment * fix: use artifact to check for ubi8 variant * chore: improve logs in resolvers * fix: apply variants to project resolver Ubi8 is a valid variant for elastic-agent * chore: use Stack version for the dependant beats In the running-on-top-of-beats feature file, the code installs some Beats and we do not want to use the given SHA, because elastic-agent now lives in a separate repo, therefore they do not share commit SHAs * fix: use fleet-ci bucket for elastic-agent and fleet-server * chore: look up both CI buckets * chore: make useCISnapshots configurable on downloads * chore: background processes should not use CI snapshots * feat: separate elastic-agent version from beats version * chore: extract useCISnapshots to a function * chore: use Github_Check_Repo to identify where the Git commit lives * fix: k8s-autodiscover lives in both worlds * fix: missing package * fix: use snapshot version in k8s-autodiscover

* ci: use withAPMEnv (#1917) * feat: partial backport for (#1628) * chore: define variable for BeatsLocalPath * fix: update path initialisation in unit tests * chore: pass inline env vars to commands Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com> Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com> * feat: export Fetch&Download methods in the /pkg directory (#1943) * chore: copy internal's version files into the exported pkg public package * chore: run unit tests from top-level goal * chore: use public package instead of the internal one * chore: remove old internal files for habdling downloads * fix: update missing references * Update `fetchBeatsBinary` to be reused in elastic-agent-poc (#1984) * update func * fix path * work on download * small fix * remove test * add sha to google * fix typo * add comment * ci: increase log rotation (#2138) * chore: increase build timeout to 90 minutes (#2139) * feat: support downloading project artifacts for the new bucket layout (#2172) * feat: support for the new bucket structure * feat: calculate bucket URLs using resolvers * chore: support for legacy Beats Old PRs needs to be tested and the binaries could not exist in the new layout * chore: support looking up the bucket in up-to 3 locations 1. new project layout 2. new beats layout 3. legacy beats layout * chore: simplify boolean logic * fix: pass worker environment for versions to the VMs * fix: predefine kibana version * fix: remove extra char * fix: remove extra char * chore: log resolvers * chore: use new project method when fetching artifacts * fix: pass stack version to stack deployment * fix: use artifact to check for ubi8 variant * chore: improve logs in resolvers * fix: apply variants to project resolver Ubi8 is a valid variant for elastic-agent * chore: use Stack version for the dependant beats In the running-on-top-of-beats feature file, the code installs some Beats and we do not want to use the given SHA, because elastic-agent now lives in a separate repo, therefore they do not share commit SHAs * fix: use fleet-ci bucket for elastic-agent and fleet-server * chore: look up both CI buckets * chore: make useCISnapshots configurable on downloads * chore: background processes should not use CI snapshots * feat: separate elastic-agent version from beats version * chore: extract useCISnapshots to a function * chore: use Github_Check_Repo to identify where the Git commit lives * fix: k8s-autodiscover lives in both worlds * fix: missing package * fix: use snapshot version in k8s-autodiscover * fix: missing import Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com> Co-authored-by: Mariana Dima <mariana@elastic.co>

* main: (268 commits) bump stack version 8.2.0-ff67d7b8 (#2242) ci: periodic builds for the last two minor versions (#2241) bump stack version 8.2.0-9bac538c (#2240) fix: add end-of-file line in templates (#2237) chore: add Christos to SSH users bump stack version 8.2.0-fee3b8d2 (#2234) Separate ES workload from Agent spec and make it req for the scenarios (#2203) bump stack version 8.2.0-d02c907a (#2228) bump stack version 8.2.0-63265ec9 (#2225) fix: wrong link (#2220) chore: define provider for each test suite (#2213) ci: provision e2-small workers (#2212) bump stack version 8.2.0-a12f7069 (#2209) chore: support for Oracle Linux 8 (#2208) bump stack version 8.2.0-190af159 (#2206) bump stack version 8.2.0-fdde08ec (#2200) bump stack version 8.2.0-43df679f (#2193) chore: use trace level in logs (#2130) bump stack version 8.2.0-bdf2ad74 (#2188) feat: support downloading project artifacts for the new bucket layout (#2172) ...

mdelapenya added 4 commits February 23, 2022 12:47

feat: support for the new bucket structure

2796cc1

feat: calculate bucket URLs using resolvers

9fc28db

chore: support for legacy Beats

8d8a301

Old PRs needs to be tested and the binaries could not exist in the new layout

chore: support looking up the bucket in up-to 3 locations

4092bc2

1. new project layout 2. new beats layout 3. legacy beats layout

mdelapenya added backport-v7.17.0 Automated backport with mergify backport-v8.0.0 Automated backport with mergify backport-v8.1.0 Automated backport with mergify labels Feb 23, 2022

mdelapenya self-assigned this Feb 23, 2022

mdelapenya requested review from a team and narph February 23, 2022 16:24

mdelapenya added area:test Anything related to the Test automation size:S less than 1 day Team:Automation Label for the Observability productivity team labels Feb 23, 2022

mdelapenya requested a review from adam-stokes February 23, 2022 16:28

narph approved these changes Feb 23, 2022

View reviewed changes

v1v approved these changes Feb 23, 2022

View reviewed changes

chore: simplify boolean logic

e89b148

v1v mentioned this pull request Feb 23, 2022

ci: refactor packaging with a new step elastic/beats#30409

Merged

adam-stokes reviewed Feb 23, 2022

View reviewed changes

adam-stokes approved these changes Feb 23, 2022

View reviewed changes

mdelapenya added 3 commits February 23, 2022 20:50

fix: pass worker environment for versions to the VMs

3fd37e3

fix: predefine kibana version

e4eafe2

fix: remove extra char

ea9f2d6

mdelapenya commented Feb 23, 2022

View reviewed changes

.ci/ansible/tasks/setup_test_script.yml Outdated Show resolved Hide resolved

fix: remove extra char

8f9fe1d

adam-stokes reviewed Feb 23, 2022

View reviewed changes

.ci/ansible/tasks/setup_test_script.yml Show resolved Hide resolved

mdelapenya added 2 commits February 24, 2022 10:44

chore: log resolvers

5548ad3

chore: use new project method when fetching artifacts

2ffd22d

mdelapenya requested a review from adam-stokes February 24, 2022 21:04

mdelapenya added 8 commits February 24, 2022 22:11

Merge branch 'main' into project-download-support

1c53d2b

Merge branch 'main' into project-download-support

bf510b3

feat: separate elastic-agent version from beats version

d0fc9ee

chore: extract useCISnapshots to a function

e08895b

chore: use Github_Check_Repo to identify where the Git commit lives

8e85453

fix: k8s-autodiscover lives in both worlds

e00336b

fix: missing package

2fe4a13

fix: use snapshot version in k8s-autodiscover

65b1761

mdelapenya mentioned this pull request Feb 28, 2022

chore: pass elastic-agent version to runE2E elastic/apm-pipeline-library#1569

Merged

adam-stokes approved these changes Feb 28, 2022

View reviewed changes

mdelapenya commented Feb 28, 2022

View reviewed changes

mdelapenya merged commit 044dedf into elastic:main Feb 28, 2022

mergify bot mentioned this pull request Feb 28, 2022

[8.1](backport #2172) feat: support downloading project artifacts for the new bucket layout #2181

Merged

This was referenced Feb 28, 2022

[8.0](backport #2172) feat: support downloading project artifacts for the new bucket layout #2182

Merged

[7.17](backport #2172) feat: support downloading project artifacts for the new bucket layout #2183

Closed

mdelapenya mentioned this pull request Feb 28, 2022

chore: 7.17 backports #2187

Merged

8 tasks

mdelapenya mentioned this pull request Mar 1, 2022

Restructure e2e suites layout #2175

Closed

mdelapenya deleted the project-download-support branch March 9, 2022 06:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: support downloading project artifacts for the new bucket layout#2172

feat: support downloading project artifacts for the new bucket layout#2172
mdelapenya merged 28 commits intoelastic:mainfrom
mdelapenya:project-download-support

mdelapenya commented Feb 23, 2022 •

edited

Loading

Uh oh!

narph left a comment

Uh oh!

elasticmachine commented Feb 23, 2022 •

edited

Loading

Build stats

Test stats 🧪

Uh oh!

adam-stokes Feb 23, 2022

Uh oh!

adam-stokes left a comment

Uh oh!

mdelapenya commented Feb 23, 2022

Uh oh!

Uh oh!

Uh oh!

mdelapenya commented Feb 24, 2022

Uh oh!

mdelapenya commented Feb 28, 2022 •

edited

Loading

Uh oh!

mdelapenya Feb 28, 2022

Uh oh!

mdelapenya Feb 28, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

mdelapenya commented Feb 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Checklist

Related issues

Other concerns

Uh oh!

narph left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Feb 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

Uh oh!

adam-stokes Feb 23, 2022

Choose a reason for hiding this comment

Uh oh!

adam-stokes left a comment

Choose a reason for hiding this comment

Uh oh!

mdelapenya commented Feb 23, 2022

Uh oh!

Uh oh!

Uh oh!

mdelapenya commented Feb 24, 2022

Uh oh!

mdelapenya commented Feb 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tests for a commit on Beats

Tests for a commit on Elastic Agent

Uh oh!

mdelapenya Feb 28, 2022

Choose a reason for hiding this comment

Uh oh!

mdelapenya Feb 28, 2022

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

mdelapenya commented Feb 23, 2022 •

edited

Loading

elasticmachine commented Feb 23, 2022 •

edited

Loading

mdelapenya commented Feb 28, 2022 •

edited

Loading