chore: bring back system integration scenarios by mdelapenya · Pull Request #2233 · elastic/e2e-testing

mdelapenya · 2022-03-14T17:53:07Z

What does this PR do?

It brings back the system_integration scenarios, which were skipped in the past with the move to the VM support.

Why is it important?

We want to double check why the data_streams are not showing the inputs.streams elements when updating a policy, which was detected while trying to restore the stand-alone mode scenarios.

We noticed that both scenarios (standalone and system integration) share the steps to update the policy, and wanted to double check that the issue that we see with standalone is not present in the system integration. Why?

after updating the package integration with system/metrics, we consider the metrics.json file is outdated
standalone mode uses docker containers, which is still reproducible locally. System metrics uses the underlying VM, so not possible to be tested locally

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have run the Unit tests (make unit-test), and they are passing locally
I have run the End-2-End tests for the suite I'm working on, and they are passing locally
I have noticed new Go dependencies (run make notice in the proper directory)

elasticmachine · 2022-03-14T18:44:20Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-03-28T15:25:41.947+0000
Duration: 72 min 1 sec

Test stats 🧪

Test	Results
Failed	0
Passed	315
Skipped	0
Total	315

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

mdelapenya · 2022-03-22T17:00:42Z

This task is stuck until we know how the metrics.json file is generated. We have contacted @IgorGuz2000 as the original author of the tests and he is researching on how it was created. Waiting for that.

mdelapenya · 2022-03-28T13:00:19Z

@IgorGuz2000 this is ready and working! Thanks for your help!!

mdelapenya · 2022-03-28T15:17:32Z

e2e/_suites/fleet/fleet.go

 	if name == "linux/metrics" {
-		file = "/linux_metrics.json"
 		metrics = "linux"
+		packageDS, err = fts.kibanaClient.GetIntegrationFromAgentPolicy(fts.currentContext, metrics, fts.Policy)


For safety, linux metrics keeps the old behaviour and reading the metrics inputs from the "linux_metrics" golden file. @IgorGuz2000 I think it could follow the same approach, although I do not see any scenario using the "linux/metrics" scenario.

Yes there is only one scenario there to verify the metrics exist

But with current project layout, I cannot see it anymore. Do you have any clue on where it is?

@IgorGuz2000 I'm going to merge this one, we can revisit the "linux_metrics" in follow ups. Thanks!

mdelapenya · 2022-03-29T08:45:46Z

@Mergifyio refresh

mdelapenya · 2022-03-29T09:03:44Z

@Mergifyio backport 7.17

mergify · 2022-03-29T09:07:07Z

refresh

✅ Pull request refreshed

* chore: bring back system integration scenarios * chore: simplify system integration scenarios to use Examples * fix: retrieve system metrics from Kibana, not from the file * fix: use example in scenario name * fix: get package policy right * chore: eval error the soonest * chore: handle not supported package policy * chore: use new method for retrieving the package policy by name * revert: log inputs from the package policy first * fix: committed by mistake * chore: retrieve package policy by ID * fix: read response properly * chore: simplify wati strategy * chore: remove unused file * chore: enable syslog integration too * chore: abstract scenario (cherry picked from commit ac1503d)

mergify · 2022-03-29T09:07:14Z

backport 7.17

✅ Backports have been created

Details

#2285 chore: bring back system integration scenarios (backport #2233) has been created for branch 7.17

* chore: bring back system integration scenarios * chore: simplify system integration scenarios to use Examples * fix: retrieve system metrics from Kibana, not from the file * fix: use example in scenario name * fix: get package policy right * chore: eval error the soonest * chore: handle not supported package policy * chore: use new method for retrieving the package policy by name * revert: log inputs from the package policy first * fix: committed by mistake * chore: retrieve package policy by ID * fix: read response properly * chore: simplify wati strategy * chore: remove unused file * chore: enable syslog integration too * chore: abstract scenario (cherry picked from commit ac1503d)

* chore: bring back system integration scenarios * chore: simplify system integration scenarios to use Examples * fix: retrieve system metrics from Kibana, not from the file * fix: use example in scenario name * fix: get package policy right * chore: eval error the soonest * chore: handle not supported package policy * chore: use new method for retrieving the package policy by name * revert: log inputs from the package policy first * fix: committed by mistake * chore: retrieve package policy by ID * fix: read response properly * chore: simplify wati strategy * chore: remove unused file * chore: enable syslog integration too * chore: abstract scenario (cherry picked from commit ac1503d) Co-authored-by: Manuel de la Peña <mdelapenya@gmail.com>

* main: (72 commits) fix: always quote variables in shell scripts (elastic#2284) chore: bring back system integration scenarios (elastic#2233) chore: do not notify build green for fleet nightly builds (elastic#2277) chore: include suite, platform and tags in test reports name (elastic#2273) bump stack version 8.2.0-dcff22d7 (elastic#2270) bump stack version 8.2.0-4509f321 (elastic#2265) chore: bump Godog to v0.12.4 (elastic#2245) bump stack version 8.2.0-5cc993c1 (elastic#2254) bump stack version 8.2.0-82b803a1 (elastic#2250) fix: never process elasticsearch CI artifacts (elastic#2246) bump stack version 8.2.0-ff67d7b8 (elastic#2242) ci: periodic builds for the last two minor versions (elastic#2241) bump stack version 8.2.0-9bac538c (elastic#2240) fix: add end-of-file line in templates (elastic#2237) chore: add Christos to SSH users bump stack version 8.2.0-fee3b8d2 (elastic#2234) Separate ES workload from Agent spec and make it req for the scenarios (elastic#2203) bump stack version 8.2.0-d02c907a (elastic#2228) bump stack version 8.2.0-63265ec9 (elastic#2225) fix: wrong link (elastic#2220) ...

chore: bring back system integration scenarios

6e4cd77

mdelapenya added Team:Automation Label for the Observability productivity team backport-v7.17.0 Automated backport with mergify backport-v8.0.0 Automated backport with mergify backport-v8.1.0 Automated backport with mergify labels Mar 14, 2022

mdelapenya self-assigned this Mar 14, 2022

mdelapenya requested review from a team, IgorGuz2000 and adam-stokes March 14, 2022 17:53

v1v approved these changes Mar 14, 2022

View reviewed changes

cachedout approved these changes Mar 15, 2022

View reviewed changes

Merge branch 'main' into bring-system-integration-back

eb5f065

cachedout added the info-needed label Mar 23, 2022

mdelapenya added 12 commits March 24, 2022 16:33

chore: simplify system integration scenarios to use Examples

d619962

fix: retrieve system metrics from Kibana, not from the file

97a0187

fix: use example in scenario name

8b8e9f5

fix: get package policy right

c6423eb

chore: eval error the soonest

deca514

chore: handle not supported package policy

8e5bbba

chore: use new method for retrieving the package policy by name

62d27c0

revert: log inputs from the package policy first

ccc1358

fix: committed by mistake

4ec68e7

chore: retrieve package policy by ID

635a286

fix: read response properly

b8d3075

chore: simplify wati strategy

009bb4c

mdelapenya marked this pull request as ready for review March 28, 2022 12:57

Merge branch 'main' into bring-system-integration-back

fa3eb53

mdelapenya added 3 commits March 28, 2022 15:56

chore: remove unused file

69a44b9

chore: enable syslog integration too

ae74e1a

chore: abstract scenario

94073d9

mdelapenya commented Mar 28, 2022

View reviewed changes

mdelapenya removed the info-needed label Mar 28, 2022

mdelapenya merged commit ac1503d into elastic:main Mar 29, 2022

mergify bot mentioned this pull request Mar 29, 2022

chore: bring back system integration scenarios (backport #2233) #2285

Merged

mergify bot mentioned this pull request Mar 29, 2022

[8.1](backport #2233) chore: bring back system integration scenarios #2286

Merged

mergify bot mentioned this pull request Mar 29, 2022

[8.0](backport #2233) chore: bring back system integration scenarios #2287

Merged

mdelapenya deleted the bring-system-integration-back branch April 11, 2022 15:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: bring back system integration scenarios#2233

chore: bring back system integration scenarios#2233
mdelapenya merged 18 commits intoelastic:mainfrom
mdelapenya:bring-system-integration-back

mdelapenya commented Mar 14, 2022

Uh oh!

elasticmachine commented Mar 14, 2022 •

edited

Loading

Build stats

Test stats 🧪

Uh oh!

mdelapenya commented Mar 22, 2022

Uh oh!

mdelapenya commented Mar 28, 2022

Uh oh!

mdelapenya Mar 28, 2022

Uh oh!

IgorGuz2000 Mar 28, 2022

Uh oh!

mdelapenya Mar 28, 2022

Uh oh!

mdelapenya Mar 29, 2022

Uh oh!

mdelapenya commented Mar 29, 2022

Uh oh!

mdelapenya commented Mar 29, 2022

Uh oh!

mergify bot commented Mar 29, 2022

Uh oh!

mergify bot commented Mar 29, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

mdelapenya commented Mar 14, 2022

What does this PR do?

Why is it important?

Checklist

Uh oh!

elasticmachine commented Mar 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

Uh oh!

mdelapenya commented Mar 22, 2022

Uh oh!

mdelapenya commented Mar 28, 2022

Uh oh!

mdelapenya Mar 28, 2022

Choose a reason for hiding this comment

Uh oh!

IgorGuz2000 Mar 28, 2022

Choose a reason for hiding this comment

Uh oh!

mdelapenya Mar 28, 2022

Choose a reason for hiding this comment

Uh oh!

mdelapenya Mar 29, 2022

Choose a reason for hiding this comment

Uh oh!

mdelapenya commented Mar 29, 2022

Uh oh!

mdelapenya commented Mar 29, 2022

Uh oh!

mergify bot commented Mar 29, 2022

✅ Pull request refreshed

Uh oh!

mergify bot commented Mar 29, 2022

✅ Backports have been created

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

elasticmachine commented Mar 14, 2022 •

edited

Loading