Adjust integration tests to make them work with fleet service token. by aleksmaus · Pull Request #1039 · elastic/fleet-server

aleksmaus · 2021-12-29T02:04:43Z

What is the problem this PR solves?

Revamp integration tests to support the username/password removal.
Addresses: #1020

How does this PR solve the problem?

Use username/password auth only for bulker integration tests and for the test setup where we need elevated permissions to be able to do perform indices setup with refresh option, in order to ensure the correct state of the index/datastream before the integration test

Here is the list of all the changes:

Update the integration token generating script to cache the token on disk, in order to allow to run individual integration tests repetitively without starting the new instance of elastic search every time. This is huge time saver during development.
Switch the tests to using existing .fleet-* indices.
Execute all integration tests serially, added -p 1 test option. This is needed now since the tests in different packages can use shared .fleet-* indices
Use username/password authenticated elastic search client for cleaning up the .fleet-* indices before running the test. This client is used for the bunker tests as well, since the tests is generic and relies on it’s own test mapping.
Move out the global checkpoint operations from the monitor package into it’s own checkpt package, and expose the API since this was needed for the actions integration tests which now doesn’t necessarily starts with the new index with -1 checkpoint value.
Remove refresh options from migration query. The fleet service token doesn't have permissions for refresh operations.

Checklist

I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

Closes Revamp integration tests to support the username/password removal #1020

mergify · 2021-12-29T02:05:16Z

This pull request does not have a backport label. Could you fix it @aleksmaus? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v/d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

aleksmaus · 2021-12-29T02:06:41Z

internal/pkg/dl/migration.go

 		client.UpdateByQuery.WithBody(reader),
 		client.UpdateByQuery.WithContext(ctx),
-		client.UpdateByQuery.WithRefresh(true),
 		client.UpdateByQuery.WithConflicts("proceed"),


had to remove

client.UpdateByQuery.WithRefresh(true)

The migration is broken without this change. The fleet service token doesn't have permissions for refresh.

The service token probably needs to have permissions for index refresh added.

Probably should comment it out with a warning rather than just delete.

yeah, will put this back if we agree to add "maintenance" permissions to the fleet-server token.
will test and add ES PR link here as a prereq.

aleksmaus · 2021-12-29T02:11:05Z

Will do another pass tomorrow and change this from draft to PR.

elasticmachine · 2021-12-29T02:12:21Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-01-03T20:01:43.957+0000
Duration: 13 min 31 sec
Commit: 7a6928d

Test stats 🧪

Test	Results
Failed	0
Passed	229
Skipped	0
Total	229

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

scunningham · 2021-12-29T14:11:40Z

internal/pkg/dl/migration.go

 		client.UpdateByQuery.WithBody(reader),
 		client.UpdateByQuery.WithContext(ctx),
-		client.UpdateByQuery.WithRefresh(true),
 		client.UpdateByQuery.WithConflicts("proceed"),


Probably should comment it out with a warning rather than just delete.

aleksmaus · 2021-12-29T16:16:28Z

Created PR for Elasticsearch elastic/elasticsearch#82125.
Can optimize the integration tests setup code once the ES change lands.

It looks like the _bulk requests still work without this permission but any other requests fail. That's why most of the code that relies on the _bulk operations still works.
Example of bulk request/response with service token:

POST /_bulk?refresh=true HTTP/1.1
Host: localhost:9200
User-Agent: go-elasticsearch/7.15.1 (darwin amd64; Go 1.17.5)
Content-Length: 323
Authorization: Bearer AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuMTpDV3RsOUtYMlFWV0x0TmtMYkpvMV9R
Content-Type: application/json
X-Elastic-Client-Meta: es=7.15.1,go=1.17.5,t=7.15.1,hc=1.17.5
X-Elastic-Product-Origin: fleet
Accept-Encoding: gzip

{"create":{"_id":"c7679bvblarusq2fasng","_index":".fleet-actions"}}
{"action_id":"ee21d4e3-9cbc-4949-b73c-7f738ff95b41","data":{"ba4cf666-696c-4660-92ba-254bbbf31c90":"582b34f1-2b31-44b7-bfb6-04501d09b478"},"expiration":"2021-12-29T14:51:05Z","input_type":"osquery","@timestamp":"2021-12-29T14:46:05Z","type":"APP_ACTION"}
HTTP/1.1 200 OK
X-elastic-product: Elasticsearch
content-type: application/json
content-encoding: gzip
content-length: 204

{"took":10,"errors":false,"items":[{"create":{"_index":".fleet-actions-7","_id":"c7679bvblarusq2fasng","_version":1,"result":"created","forced_refresh":true,"_shards":{"total":1,"successful":1,"failed":0},"_seq_no":42,"_primary_term":1,"status":201}}]}

This seems to be a bug on Elasticsearch side. If it gets fixed it can break fleet server unless we add the permission that allows fleet server refresh ops.

aleksmaus · 2021-12-30T03:11:42Z

Updated integration tests setup to use the token for setup.
Rolled back the change in the migration call, did put the refresh back (@scunningham).
Tested locally against Elasticsearch with the fleet server token permissions updated.

This PR will fail in CI for now until the Elasticsearch image with the permissions change is published and picked up for our integration tests.

… boostrap indicies/datastreams appropriately.

scunningham

Nice reWamp.

aleksmaus · 2021-12-30T16:52:21Z

Nice reWamp.

…1039) * Adjust integration tests to make them work with fleet service token. * Remove .fleet-* mapping cruft. Test rely on .fleet indices plugin to bootstrap indices/datastreams appropriately. * Update integration tests setup to use the token for setup (cherry picked from commit aead51b)

…1039) (#1058) * Adjust integration tests to make them work with fleet service token. * Remove .fleet-* mapping cruft. Test rely on .fleet indices plugin to bootstrap indices/datastreams appropriately. * Update integration tests setup to use the token for setup (cherry picked from commit aead51b) Co-authored-by: Aleksandr Maus <aleksandr.maus@elastic.co> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

aleksmaus added the cleanup label Dec 29, 2021

aleksmaus requested review from blakerouse and scunningham December 29, 2021 02:04

mergify bot added the backport-skip Skip notification from the automated backport with mergify label Dec 29, 2021

aleksmaus commented Dec 29, 2021

View reviewed changes

aleksmaus added backport-v8.0.0 Automated backport with mergify backport-v8.1.0 Automated backport with mergify and removed backport-skip Skip notification from the automated backport with mergify labels Dec 29, 2021

aleksmaus force-pushed the rewamp/integration_tests branch 2 times, most recently from 1969f58 to 1006f0b Compare December 29, 2021 03:02

scunningham suggested changes Dec 29, 2021

View reviewed changes

aleksmaus mentioned this pull request Dec 29, 2021

Add "maintenance" permission to the fleet-server service account elastic/elasticsearch#82125

Merged

aleksmaus marked this pull request as ready for review December 29, 2021 16:16

aleksmaus requested review from lykkin, michel-laterman and ruflin December 29, 2021 16:16

aleksmaus requested a review from scunningham December 30, 2021 03:11

aleksmaus added 3 commits December 30, 2021 09:30

Adjust integration tests to make them work with fleet service token.

a214657

Remove .fleet-* mapping cruft. Test rely on .fleet indicies plugin to…

5bea285

… boostrap indicies/datastreams appropriately.

Update integration tests setup to use the token for setup

c3cfd0f

aleksmaus force-pushed the rewamp/integration_tests branch from 0db444e to c3cfd0f Compare December 30, 2021 14:30

scunningham approved these changes Dec 30, 2021

View reviewed changes

aleksmaus added 2 commits January 3, 2022 08:40

Merge branch 'master' into rewamp/integration_tests

5ace7ce

Make check happy

7a6928d

aleksmaus merged commit aead51b into elastic:master Jan 4, 2022

mergify bot mentioned this pull request Jan 4, 2022

[8.0](backport #1039) Adjust integration tests to make them work with fleet service token. #1058

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adjust integration tests to make them work with fleet service token.#1039

Adjust integration tests to make them work with fleet service token.#1039
aleksmaus merged 5 commits intoelastic:masterfrom
aleksmaus:rewamp/integration_tests

aleksmaus commented Dec 29, 2021

Uh oh!

mergify bot commented Dec 29, 2021

Uh oh!

aleksmaus Dec 29, 2021 •

edited

Loading

Uh oh!

scunningham Dec 29, 2021

Uh oh!

aleksmaus Dec 29, 2021 •

edited

Loading

Uh oh!

aleksmaus commented Dec 29, 2021

Uh oh!

elasticmachine commented Dec 29, 2021 •

edited

Loading

Build stats

Test stats 🧪

Uh oh!

scunningham Dec 29, 2021

Uh oh!

aleksmaus commented Dec 29, 2021

Uh oh!

aleksmaus commented Dec 30, 2021

Uh oh!

scunningham left a comment

Uh oh!

aleksmaus commented Dec 30, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

aleksmaus commented Dec 29, 2021

What is the problem this PR solves?

How does this PR solve the problem?

Checklist

Related issues

Uh oh!

mergify bot commented Dec 29, 2021

Uh oh!

aleksmaus Dec 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

scunningham Dec 29, 2021

Choose a reason for hiding this comment

Uh oh!

aleksmaus Dec 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aleksmaus commented Dec 29, 2021

Uh oh!

elasticmachine commented Dec 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

Uh oh!

scunningham Dec 29, 2021

Choose a reason for hiding this comment

Uh oh!

aleksmaus commented Dec 29, 2021

Uh oh!

aleksmaus commented Dec 30, 2021

Uh oh!

scunningham left a comment

Choose a reason for hiding this comment

Uh oh!

aleksmaus commented Dec 30, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

aleksmaus Dec 29, 2021 •

edited

Loading

aleksmaus Dec 29, 2021 •

edited

Loading

elasticmachine commented Dec 29, 2021 •

edited

Loading