Skip to content
This repository was archived by the owner on Sep 17, 2024. It is now read-only.

v2 refactor (backport #1008)#1069

Merged
adam-stokes merged 1 commit into7.xfrom
mergify/bp/7.x/pr-1008
Apr 21, 2021
Merged

v2 refactor (backport #1008)#1069
adam-stokes merged 1 commit into7.xfrom
mergify/bp/7.x/pr-1008

Conversation

@mergify
Copy link
Copy Markdown
Contributor

@mergify mergify bot commented Apr 21, 2021

This is an automatic backport of pull request #1008 done by Mergify.

Cherry-pick of 5f59670 has failed:

On branch mergify/bp/7.x/pr-1008
Your branch is up to date with 'origin/7.x'.

You are currently cherry-picking commit 5f596709.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   cli/cmd/deploy.go
	modified:   cli/cmd/run.go
	modified:   cli/cmd/stop.go
	modified:   cli/cmd/sync.go
	modified:   cli/cmd/sync_test.go
	modified:   cli/config/config.go
	modified:   cli/config/config_test.go
	deleted:    cli/services/kibana.go
	deleted:    cli/services/kibana_test.go
	modified:   e2e/_suites/fleet/features/agent_endpoint_integration.feature
	modified:   e2e/_suites/fleet/features/backend_processes.feature
	modified:   e2e/_suites/fleet/fleet.go
	modified:   e2e/_suites/fleet/fleet_server.go
	deleted:    e2e/_suites/fleet/installers.go
	deleted:    e2e/_suites/fleet/integrations.go
	deleted:    e2e/_suites/fleet/services.go
	modified:   e2e/_suites/helm/helm_charts_test.go
	modified:   e2e/_suites/metricbeat/metricbeat_test.go
	modified:   e2e/steps/befores.go
	modified:   e2e/steps/configurations.go
	deleted:    e2e/steps/processes.go
	deleted:    e2e/steps/services.go
	new file:   internal/common/defaults.go
	new file:   internal/common/retry.go
	renamed:    cli/services/manager.go -> internal/compose/compose.go
	renamed:    cli/shell/curl.go -> internal/curl/curl.go
	renamed:    cli/docker/docker.go -> internal/docker/docker.go
	renamed:    e2e/assertions.go -> internal/elasticsearch/assertions.go
	renamed:    e2e/elasticsearch.go -> internal/elasticsearch/client.go
	new file:   internal/git/git.go
	new file:   internal/git/git_test.go
	renamed:    cli/services/helm.go -> internal/helm/helm.go
	new file:   internal/installer/base.go
	renamed:    e2e/_suites/fleet/services_test.go -> internal/installer/base_test.go
	new file:   internal/installer/deb.go
	new file:   internal/installer/docker.go
	new file:   internal/installer/elasticagent.go
	new file:   internal/installer/rpm.go
	new file:   internal/installer/tar.go
	new file:   internal/io/io.go
	new file:   internal/io/io_test.go
	new file:   internal/kibana/agents.go
	new file:   internal/kibana/client.go
	new file:   internal/kibana/client_test.go
	new file:   internal/kibana/fleet.go
	new file:   internal/kibana/integrations.go
	new file:   internal/kibana/policies.go
	new file:   internal/kibana/server.go
	new file:   internal/kibana/url_prefixes.go
	renamed:    cli/services/kubectl.go -> internal/kubectl/kubectl.go
	renamed:    cli/services/sanitizer.go -> internal/sanitizer/sanitizer.go
	renamed:    cli/services/sanitizer_test.go -> internal/sanitizer/sanitizer_test.go
	renamed:    cli/shell/shell.go -> internal/shell/shell.go
	renamed:    cli/shell/shell_test.go -> internal/shell/shell_test.go
	new file:   internal/state/state.go
	new file:   internal/state/state_test.go
	renamed:    e2e/utils.go -> internal/utils/utils.go
	renamed:    e2e/utils_test.go -> internal/utils/utils_test.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   e2e/Makefile
	both modified:   e2e/_suites/fleet/ingest_manager_test.go
	both modified:   e2e/_suites/fleet/stand-alone.go
	both modified:   e2e/_suites/fleet/world.go

To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/


Mergify commands and options

More conditions and actions can be found in the documentation.

You can also trigger Mergify actions by commenting on this pull request:

  • @Mergifyio refresh will re-evaluate the rules
  • @Mergifyio rebase will rebase this PR on its base branch
  • @Mergifyio update will merge the base branch into this PR
  • @Mergifyio backport <destination> will backport this PR on <destination> branch

Additionally, on Mergify dashboard you can:

  • look at your merge queues
  • generate the Mergify configuration with the config editor.

Finally, you can contact us on https://mergify.io/

* Move kibana into internals, update fleet test suite
* migrate docker-compose related code to internal layout
* move docker related code to internal layout
* move git related code to internal layout
* move common attributes into internal common file system layout
* move elasticsearch specifics into its own filesystem layout
* move installer based code to internal layout
* move shell related code to internal layout
* move sanitizer code to internal layout
* move io related code to internal layout
* move utils into internal layout
* Update package integration querying/altering
* move curl to internal layout
* move helm to internal layout
* move kubectl into internal layout
* move state internal filesystem
* cleanup config in stand-alone
* remove unused files
* Uniquify the stand-alone step for checking agent status (#993)

There were 2 steps identical in both the stand-alone and fleet test suites.
Running the stand-alone test suite was picking up the step from the fleet test
suite and trying to reference the FleetTestSuite structure which did not hold
any of the agent information (like the hostname) for the stand alone tests.

This fixes it so that the standalone test step is being referenced in the
correct test suite.

* Remove the agent config file parameters for stand alone (#983)
* Update helm/metricbeat tests to use new layout
* Fix policy endpoint update
* fix panic on helm init
* Fix step reference as this being merged seperately
* Update function call to correct standalone step
* Fix merge conflict
* update ProfileEnv query/set for KibanaVersion
* More fixes to agent endpoint security checks
* update backend feature to call out endpoint in step
* use common.TimeoutFactor in docker checkprocess state
* Update adding endpoint integration
* enable features for fleet server
* not necessary to enroll after install
* wait for filebeat/metricbeat before restarts
* clear out fts.CurrentToken during beforeScenario
* attach system integration on deploy
* enroll if rpm
* dont store fleet policy
* update kibana config for latest fleet server
* Update e2e/_suites/fleet/fleet.go
* Update e2e/_suites/fleet/fleet.go
* Update e2e/_suites/fleet/fleet.go
* Update .pre-commit-config.yaml
* Update e2e/Makefile
* rename apt -> deb for installer type
* execute docker start/stop with timeout between
* fixes fleet_server scenario
* Utilize fleet server in all tests
* Fix enrollment url for fleet server
* Query elasticsearch logs for endpoint security event changes
* Increase search result size for ES
* Fix issue with fleet server restarting continuously
* unpin kibana pr now that most major breakage is resolved
* force unenroll
* for new fleet bootstrap on re-enrollment
* Fix unenrollment
* Add timeout safeguard to elastic-agent execution

In some cases such as attempting to re-enroll with a revoked token, the
elastic-agent will retry indefinitely. This fix adds a safeguard utilizing
'timeout' command prepended to the elastic-agent command so that it will timeout
after TimeoutFactor

Signed-off-by: Adam Stokes <51892+adam-stokes@users.noreply.github.com>
Co-authored-by: Manuel de la Peña <mdelapenya@gmail.com>
(cherry picked from commit 5f59670)

# Conflicts:
#	e2e/Makefile
#	e2e/_suites/fleet/ingest_manager_test.go
#	e2e/_suites/fleet/stand-alone.go
#	e2e/_suites/fleet/world.go
@mergify mergify bot added the conflicts There is a conflict in the backported pull request label Apr 21, 2021
@elasticmachine
Copy link
Copy Markdown
Contributor

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #1069 opened

  • Start Time: 2021-04-21T17:29:39.550+0000

  • Duration: 5 min 37 sec

  • Commit: 69fa7cf

Test stats 🧪

Test Results
Failed 0
Passed 40
Skipped 0
Total 40

Trends 🧪

Image of Build Times

Image of Tests

Steps errors 2

Expand to view the steps failures

Build and test
  • Took 0 min 25 sec . View more details on here
  • Description: .ci/scripts/build-test.sh
Archive the artifacts
  • Took 0 min 0 sec . View more details on here
  • Description: [2021-04-21T17:32:20.309Z] Archiving artifacts script returned exit code 2

Log output

Expand to view the last 100 lines of log output

[2021-04-21T17:33:19.895Z] Installing setuptools, pip, wheel...
[2021-04-21T17:33:23.188Z] done.
[2021-04-21T17:33:24.127Z] Collecting pre-commit
[2021-04-21T17:33:24.127Z]   Downloading pre_commit-2.12.1-py2.py3-none-any.whl (189 kB)
[2021-04-21T17:33:24.127Z] Collecting importlib-resources
[2021-04-21T17:33:24.127Z]   Downloading importlib_resources-5.1.2-py3-none-any.whl (25 kB)
[2021-04-21T17:33:24.385Z] Collecting toml
[2021-04-21T17:33:24.385Z]   Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB)
[2021-04-21T17:33:24.385Z] Collecting nodeenv>=0.11.1
[2021-04-21T17:33:24.385Z]   Downloading nodeenv-1.6.0-py2.py3-none-any.whl (21 kB)
[2021-04-21T17:33:24.645Z] Collecting virtualenv>=20.0.8
[2021-04-21T17:33:24.645Z]   Downloading virtualenv-20.4.4-py2.py3-none-any.whl (7.2 MB)
[2021-04-21T17:33:25.213Z] Collecting identify>=1.0.0
[2021-04-21T17:33:25.213Z]   Downloading identify-2.2.4-py2.py3-none-any.whl (98 kB)
[2021-04-21T17:33:25.473Z] Collecting importlib-metadata
[2021-04-21T17:33:25.473Z]   Downloading importlib_metadata-4.0.1-py3-none-any.whl (16 kB)
[2021-04-21T17:33:25.473Z] Collecting cfgv>=2.0.0
[2021-04-21T17:33:25.473Z]   Downloading cfgv-3.2.0-py2.py3-none-any.whl (7.3 kB)
[2021-04-21T17:33:25.473Z] Collecting pyyaml>=5.1
[2021-04-21T17:33:25.473Z]   Downloading PyYAML-5.4.1-cp36-cp36m-manylinux1_x86_64.whl (640 kB)
[2021-04-21T17:33:25.732Z] Collecting appdirs<2,>=1.4.3
[2021-04-21T17:33:25.732Z]   Downloading appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
[2021-04-21T17:33:25.732Z] Collecting distlib<1,>=0.3.1
[2021-04-21T17:33:25.732Z]   Downloading distlib-0.3.1-py2.py3-none-any.whl (335 kB)
[2021-04-21T17:33:25.732Z] Collecting six<2,>=1.9.0
[2021-04-21T17:33:25.732Z]   Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
[2021-04-21T17:33:25.992Z] Collecting filelock<4,>=3.0.0
[2021-04-21T17:33:25.992Z]   Downloading filelock-3.0.12-py3-none-any.whl (7.6 kB)
[2021-04-21T17:33:25.992Z] Collecting zipp>=0.5
[2021-04-21T17:33:25.992Z]   Downloading zipp-3.4.1-py3-none-any.whl (5.2 kB)
[2021-04-21T17:33:25.992Z] Collecting typing-extensions>=3.6.4
[2021-04-21T17:33:26.253Z]   Downloading typing_extensions-3.7.4.3-py3-none-any.whl (22 kB)
[2021-04-21T17:33:26.253Z] Installing collected packages: zipp, typing-extensions, six, importlib-resources, importlib-metadata, filelock, distlib, appdirs, virtualenv, toml, pyyaml, nodeenv, identify, cfgv, pre-commit
[2021-04-21T17:33:26.822Z] Successfully installed appdirs-1.4.4 cfgv-3.2.0 distlib-0.3.1 filelock-3.0.12 identify-2.2.4 importlib-metadata-4.0.1 importlib-resources-5.1.2 nodeenv-1.6.0 pre-commit-2.12.1 pyyaml-5.4.1 six-1.15.0 toml-0.10.2 typing-extensions-3.7.4.3 virtualenv-20.4.4 zipp-3.4.1
[2021-04-21T17:33:26.822Z] Downloading https://files.pythonhosted.org/packages/a4/e3/1f067de470e3a86875ed915438dc3bd781fb0346254f541190a09472b677/virtualenv-16.7.10.tar.gz
[2021-04-21T17:33:26.822Z] Done.
[2021-04-21T17:33:26.822Z] *******************************************************************************
[2021-04-21T17:33:26.822Z] Installing pre-commit to /var/lib/jenkins/workspace/2e-tests_e2e-testing-mbp_PR-1069/bin/pre-commit
[2021-04-21T17:33:26.822Z] *******************************************************************************
[2021-04-21T17:33:27.585Z] + export PATH=/var/lib/jenkins/workspace/2e-tests_e2e-testing-mbp_PR-1069/bin:/var/lib/jenkins/workspace/2e-tests_e2e-testing-mbp_PR-1069/bin:/var/lib/jenkins/workspace/2e-tests_e2e-testing-mbp_PR-1069/.gvm/versions/go1.14.12.linux.amd64/bin:/var/lib/jenkins/workspace/2e-tests_e2e-testing-mbp_PR-1069/src/github.com/elastic/e2e-testing/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/lib/jenkins/workspace/2e-tests_e2e-testing-mbp_PR-1069/bin:/var/lib/jenkins/workspace/2e-tests_e2e-testing-mbp_PR-1069/src/github.com/elastic/e2e-testing/.ci/scripts
[2021-04-21T17:33:27.585Z] + pre-commit install --install-hooks
[2021-04-21T17:33:27.585Z] pre-commit installed at .git/hooks/pre-commit
[2021-04-21T17:33:27.585Z] [WARNING] The 'rev' field of repo 'git@github.com:elastic/apm-pipeline-library' appears to be a mutable reference (moving tag / branch).  Mutable references are never updated after first install and are not supported.  See https://pre-commit.com/#using-the-latest-version-for-a-repository for more details.  Hint: `pre-commit autoupdate` often fixes this.
[2021-04-21T17:33:27.585Z] [INFO] Initializing environment for git://github.com/pre-commit/pre-commit-hooks.
[2021-04-21T17:33:28.153Z] [INFO] Initializing environment for git://github.com/dnephin/pre-commit-golang.
[2021-04-21T17:33:28.413Z] [INFO] Initializing environment for git@github.com:elastic/apm-pipeline-library.
[2021-04-21T17:33:29.792Z] [INFO] Installing environment for git://github.com/pre-commit/pre-commit-hooks.
[2021-04-21T17:33:29.792Z] [INFO] Once installed this environment will be reused.
[2021-04-21T17:33:29.792Z] [INFO] This may take a few minutes...
[2021-04-21T17:33:35.429Z] + export PATH=/var/lib/jenkins/workspace/2e-tests_e2e-testing-mbp_PR-1069/bin:/var/lib/jenkins/workspace/2e-tests_e2e-testing-mbp_PR-1069/bin:/var/lib/jenkins/workspace/2e-tests_e2e-testing-mbp_PR-1069/.gvm/versions/go1.14.12.linux.amd64/bin:/var/lib/jenkins/workspace/2e-tests_e2e-testing-mbp_PR-1069/src/github.com/elastic/e2e-testing/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/lib/jenkins/workspace/2e-tests_e2e-testing-mbp_PR-1069/bin:/var/lib/jenkins/workspace/2e-tests_e2e-testing-mbp_PR-1069/src/github.com/elastic/e2e-testing/.ci/scripts
[2021-04-21T17:33:35.429Z] + set +e
[2021-04-21T17:33:35.429Z] + find /var/lib/jenkins/workspace/2e-tests_e2e-testing-mbp_PR-1069/.cache/pre-commit -type d -name scripts
[2021-04-21T17:33:35.429Z] + grep .ci/scripts
[2021-04-21T17:33:35.430Z] + searchFile=/var/lib/jenkins/workspace/2e-tests_e2e-testing-mbp_PR-1069/.cache/pre-commit/repou1bcx8vj/.ci/scripts
[2021-04-21T17:33:35.430Z] + [ -e /var/lib/jenkins/workspace/2e-tests_e2e-testing-mbp_PR-1069/.cache/pre-commit/repou1bcx8vj/.ci/scripts ]
[2021-04-21T17:33:35.430Z] + export PATH=/var/lib/jenkins/workspace/2e-tests_e2e-testing-mbp_PR-1069/bin:/var/lib/jenkins/workspace/2e-tests_e2e-testing-mbp_PR-1069/bin:/var/lib/jenkins/workspace/2e-tests_e2e-testing-mbp_PR-1069/.gvm/versions/go1.14.12.linux.amd64/bin:/var/lib/jenkins/workspace/2e-tests_e2e-testing-mbp_PR-1069/src/github.com/elastic/e2e-testing/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/lib/jenkins/workspace/2e-tests_e2e-testing-mbp_PR-1069/bin:/var/lib/jenkins/workspace/2e-tests_e2e-testing-mbp_PR-1069/src/github.com/elastic/e2e-testing/.ci/scripts:/var/lib/jenkins/workspace/2e-tests_e2e-testing-mbp_PR-1069/.cache/pre-commit/repou1bcx8vj/.ci/scripts
[2021-04-21T17:33:35.430Z] + set -e
[2021-04-21T17:33:35.430Z] + git diff-tree --no-commit-id --name-only -r 69fa7cf93359d7ae8f8277596604a1b50221404e
[2021-04-21T17:33:35.430Z] + + xargs pre-committee run pre-commit.out --files
[2021-04-21T17:33:35.430Z] 
[2021-04-21T17:33:35.430Z] [WARNING] The 'rev' field of repo 'git@github.com:elastic/apm-pipeline-library' appears to be a mutable reference (moving tag / branch).  Mutable references are never updated after first install and are not supported.  See https://pre-commit.com/#using-the-latest-version-for-a-repository for more details.  Hint: `pre-commit autoupdate` often fixes this.
[2021-04-21T17:33:35.689Z] Check for case conflicts.................................................Passed
[2021-04-21T17:33:35.689Z] Check that executables have shebangs.................(no files to check)Skipped
[2021-04-21T17:33:35.689Z] Check JSON...........................................(no files to check)Skipped
[2021-04-21T17:33:35.956Z] Check for merge conflicts................................................Passed
[2021-04-21T17:33:35.956Z] Check Yaml...........................................(no files to check)Skipped
[2021-04-21T17:33:35.956Z] Check Xml............................................(no files to check)Skipped
[2021-04-21T17:33:35.956Z] Fix End of Files.........................................................Passed
[2021-04-21T17:33:35.956Z] go fmt...................................................................Passed
[2021-04-21T17:33:36.216Z] go lint..................................................................Passed
[2021-04-21T17:33:36.216Z] Check files aren't using go's testing package........(no files to check)Skipped
[2021-04-21T17:33:54.326Z] golangci-lint............................................................Passed
[2021-04-21T17:33:54.326Z] check-bash-syntax....................................(no files to check)Skipped
[2021-04-21T17:33:54.326Z] check-jenkins-pipelines..............................(no files to check)Skipped
[2021-04-21T17:33:54.326Z] - hook id: check-jenkins-pipelines
[2021-04-21T17:33:54.326Z] check-unicode-non-breaking-spaces........................................Passed
[2021-04-21T17:33:54.326Z] remove-unicode-non-breaking-spaces.......................................Passed
[2021-04-21T17:33:54.326Z] check-en-dashes..........................................................Passed
[2021-04-21T17:33:54.326Z] remove-en-dashes.........................................................Passed
[2021-04-21T17:33:54.326Z] check-jjbb...........................................(no files to check)Skipped
[2021-04-21T17:33:54.326Z] - hook id: check-jjbb
[2021-04-21T17:34:12.429Z] check-gherkin-lint.......................................................Passed
[2021-04-21T17:34:12.502Z] $ ssh-agent -k
[2021-04-21T17:34:12.506Z] unset SSH_AUTH_SOCK;
[2021-04-21T17:34:12.507Z] unset SSH_AGENT_PID;
[2021-04-21T17:34:12.507Z] echo Agent pid 4940 killed;
[2021-04-21T17:34:12.517Z] [ssh-agent] Stopped.
[2021-04-21T17:34:12.706Z] Archiving artifacts
[2021-04-21T17:34:12.883Z] Recording test results
[2021-04-21T17:34:13.129Z] [Checks API] No suitable checks publisher found.
[2021-04-21T17:34:14.648Z] Stage "Build Docs" skipped due to earlier failure(s)
[2021-04-21T17:34:14.739Z] Stage "End-To-End Tests" skipped due to earlier failure(s)
[2021-04-21T17:34:14.826Z] Stage "Release" skipped due to earlier failure(s)
[2021-04-21T17:34:15.293Z] Running on Jenkins in /var/lib/jenkins/workspace/2e-tests_e2e-testing-mbp_PR-1069
[2021-04-21T17:34:15.391Z] [INFO] getVaultSecret: Getting secrets
[2021-04-21T17:34:15.497Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-04-21T17:34:16.434Z] + chmod 755 generate-build-data.sh
[2021-04-21T17:34:16.434Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/e2e-tests/e2e-testing-mbp/PR-1069/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/e2e-tests/e2e-testing-mbp/PR-1069/runs/1 FAILURE 276613
[2021-04-21T17:34:16.684Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/e2e-tests/e2e-testing-mbp/PR-1069/runs/1/steps/?limit=10000 -o steps-info.json
[2021-04-21T17:34:17.596Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/e2e-tests/e2e-testing-mbp/PR-1069/runs/1/tests/?status=FAILED -o tests-errors.json

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 40
Skipped 0
Total 40

@adam-stokes adam-stokes merged commit d757abf into 7.x Apr 21, 2021
@mergify mergify bot deleted the mergify/bp/7.x/pr-1008 branch April 21, 2021 17:46
adam-stokes added a commit that referenced this pull request Apr 21, 2021
This reverts commit d757abf.

Signed-off-by: Adam Stokes <51892+adam-stokes@users.noreply.github.com>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

conflicts There is a conflict in the backported pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants