[Security solution] [Endpoint] Fix bad artifact migration by dasansol92 · Pull Request #111294 · elastic/kibana

dasansol92 · 2021-09-06T15:16:44Z

Summary

Zip compression was removed during artifacts generation as it was done on the fleet side.
We need to keep this behaviour for the migration script as it is taking the old ones already zip compressed.

fixes #111210

For maintainers

This was checked for breaking API changes and was labeled appropriately

…O artifact

…e creating the new one from fleet

…tArtifact and deleteArtifact as there is an ES error in those methods if not mocked

dasansol92 · 2021-09-07T09:45:34Z

@elasticmachine merge upstream

elasticmachine · 2021-09-07T12:26:43Z

Pinging @elastic/security-onboarding-and-lifecycle-mgt (Team:Onboarding and Lifecycle Mgt)

paul-tavares · 2021-09-07T12:31:51Z

x-pack/plugins/fleet/server/services/artifacts/mocks.ts

 };

-export const generateArtifactEsGetSingleHitMock = (): SearchHit<ArtifactElasticsearchProperties> => {
+export const generateArtifactEsGetSingleHitMock = (


FYI: Just making this mock provider a bit more robust by accepting on input an optional artifact

elasticmachine · 2021-09-07T12:31:55Z

Pinging @elastic/fleet (Team:Fleet)

paul-tavares

🔥 🔥

academo · 2021-09-07T12:38:33Z

x-pack/plugins/security_solution/server/endpoint/lib/artifacts/migrate_artifacts_to_fleet.ts

+          artifact.attributes.body = (
+            await inflateAsync(Buffer.from(artifact.attributes.body, 'base64'))
+          ).toString('base64');
+        }


is there a case where is not compressed? if so, do we have tests to cover it?

@paul-tavares correct me if I'm wrong, but if this script is executed just one time it shouldn't be non compressed artifacts right?

@academo no, there is no case where an Artifact would be stored in the Saved Object with out being compressed. I think this check is just to be safe, but agree is not technically needed.

joshdover

Changes to Fleet mock LGTM

paul-tavares · 2021-09-07T14:13:39Z

...k/plugins/security_solution/server/endpoint/lib/artifacts/migrate_artifacts_to_fleet.test.ts

+      expect.objectContaining({
+        ...soArtifactEntry,
+        compressionAlgorithm: 'zlib',
+        body: 'eJyrVkrNKynKTC1WsoqOrQUAJxkFKQ==',


Just noticed this (thanks for pointing it out.
Can you remove this and instead changing the migration change to not mutate the body prop? like:

if (isCompressed(artifact.attributes)) { artifact.attributes = { ...artifact.attributes, body: ( await inflateAsync(Buffer.from(artifact.attributes.body, 'base64')) ).toString('base64'), }; }

Or - capture a copy of the soArtifatEntry here in the test prior to calling migration (above like 115) and then use that to check again on like 119 (and remove body from here)

paul-tavares · 2021-09-07T14:17:01Z

x-pack/plugins/security_solution/server/endpoint/lib/artifacts/migrate_artifacts_to_fleet.ts

      for (const artifact of artifactList) {
+        if (isCompressed(artifact.attributes)) {
+          artifact.attributes = {
+            ...artifact.attributes,


paul-tavares

🚢

kibanamachine · 2021-09-07T16:35:18Z

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

💛 Build #151426 was flaky 36fa4cd
💔 Build #151405 failed fe33bb3
💔 Build #151324 failed 77358c7

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @paul-tavares

…1294) * Working test that validate migrated artifact has same properties as SO artifact * Checks if artifact is compressed and uncompress it if necessary before creating the new one from fleet Co-authored-by: Paul Tavares <paul.tavares@elastic.co> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

kibanamachine · 2021-09-07T16:39:40Z

💚 Backport successful

Status	Branch	Result
✅	7.15
✅	7.x
✅	7.14

The backport PRs will be merged automatically after passing CI.

…111432) * Working test that validate migrated artifact has same properties as SO artifact * Checks if artifact is compressed and uncompress it if necessary before creating the new one from fleet Co-authored-by: Paul Tavares <paul.tavares@elastic.co> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: David Sánchez <davidsansol92@gmail.com> Co-authored-by: Paul Tavares <paul.tavares@elastic.co>

…111430) * Working test that validate migrated artifact has same properties as SO artifact * Checks if artifact is compressed and uncompress it if necessary before creating the new one from fleet Co-authored-by: Paul Tavares <paul.tavares@elastic.co> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: David Sánchez <davidsansol92@gmail.com> Co-authored-by: Paul Tavares <paul.tavares@elastic.co>

…111431) * Working test that validate migrated artifact has same properties as SO artifact * Checks if artifact is compressed and uncompress it if necessary before creating the new one from fleet Co-authored-by: Paul Tavares <paul.tavares@elastic.co> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: David Sánchez <davidsansol92@gmail.com> Co-authored-by: Paul Tavares <paul.tavares@elastic.co>

…-link-to-kibana-app * 'master' of github.com:elastic/kibana: (61 commits) [Logs UI] Fix alert previews for thresholds of `0` (elastic#111150) [Archive Migration][Partial] discover apps-discover (elastic#110437) [APM] Set start date of APM ML job to -4 weeks (elastic#111375) [ML] APM Latency Correlations: Code consolidation. (elastic#110790) [Discover] Fix indices permission for multiline test (elastic#111284) [Detection Rules] Add 7.15 rules (elastic#111464) [Security Solution][Endpoint][Host Isolation] Hide isolate host option in alert details rather than disabling (elastic#111064) React version of angular license view (elastic#111317) [APM] Fix link in readme (elastic#111362) [Security Solution] add agent field to generator (elastic#111428) [Dashboard] Retain Tags on Quicksave (elastic#111015) Reorder App Search ingestion methods (elastic#111361) Port performance docs to new docs system. (elastic#111063) [Security Solution][RAC] Fixes updatedAt loading bug (elastic#111010) [sample data] update web log geo.src field to match country code of geo.coordinates (elastic#110885) [Security solution] [Endpoint] Fix bad artifact migration (elastic#111294) Fix copy typo. (elastic#111203) [build] Remove empty optimize directory (elastic#111393) [Maps] fix term join not updating when editing right field (elastic#111030) [Fleet] Set default settings in component template instead of the index template (elastic#111197) ... # Conflicts: # x-pack/plugins/reporting/public/management/__snapshots__/report_listing.test.tsx.snap # x-pack/plugins/reporting/public/management/report_listing.test.tsx

…1294) * Working test that validate migrated artifact has same properties as SO artifact * Checks if artifact is compressed and uncompress it if necessary before creating the new one from fleet Co-authored-by: Paul Tavares <paul.tavares@elastic.co> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

paul-tavares and others added 4 commits September 3, 2021 17:56

attempt to get test done for bad migration (still not working)

8bfe120

Working test that validate migrated artifact has same properties as S…

c92b8ff

…O artifact

Checks if artifact is compressed and uncompress it if necessary befor…

77358c7

…e creating the new one from fleet

Use non mocked instance for createArtifact and mocked instance for ge…

fe33bb3

…tArtifact and deleteArtifact as there is an ES error in those methods if not mocked

Merge branch 'master' into fix/olm-kibana-111210-bad-artifact-migration

36fa4cd

dasansol92 marked this pull request as ready for review September 7, 2021 12:26

dasansol92 requested review from a team as code owners September 7, 2021 12:26

dasansol92 added auto-backport Deprecated - use backport:version if exact versions are needed release_note:fix Team:Defend Workflows “EDR Workflows” sub-team of Security Solution v8.0.0 labels Sep 7, 2021

dasansol92 requested review from a team, academo and paul-tavares and removed request for a team September 7, 2021 12:27

paul-tavares added the v7.16.0 label Sep 7, 2021

paul-tavares reviewed Sep 7, 2021

View reviewed changes

botelastic bot added the Team:Fleet Team label for Observability Data Collection Fleet team label Sep 7, 2021

paul-tavares approved these changes Sep 7, 2021

View reviewed changes

academo reviewed Sep 7, 2021

View reviewed changes

dasansol92 added v7.14.2 v7.15.0 labels Sep 7, 2021

joshdover approved these changes Sep 7, 2021

View reviewed changes

paul-tavares reviewed Sep 7, 2021

View reviewed changes

Don't mutate data in script

6e13304

paul-tavares reviewed Sep 7, 2021

View reviewed changes

paul-tavares approved these changes Sep 7, 2021

View reviewed changes

academo approved these changes Sep 7, 2021

View reviewed changes

paul-tavares self-assigned this Sep 7, 2021

paul-tavares merged commit 855a338 into elastic:master Sep 7, 2021

kibanamachine mentioned this pull request Sep 7, 2021

[7.15] [Security solution] [Endpoint] Fix bad artifact migration (#111294) #111430

Merged

kibanamachine mentioned this pull request Sep 7, 2021

[7.x] [Security solution] [Endpoint] Fix bad artifact migration (#111294) #111431

Merged

kibanamachine mentioned this pull request Sep 7, 2021

[7.14] [Security solution] [Endpoint] Fix bad artifact migration (#111294) #111432

Merged

kevinlog mentioned this pull request Sep 9, 2021

[Agents]: Artifacts related and "Initial configuration application failed" errors on adding Endpoint Security to the Agent policy. elastic/beats#27064

Closed

Conversation

dasansol92 commented Sep 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

For maintainers

Uh oh!

dasansol92 commented Sep 7, 2021

Uh oh!

elasticmachine commented Sep 7, 2021

Uh oh!

paul-tavares Sep 7, 2021

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Sep 7, 2021

Uh oh!

paul-tavares left a comment

Choose a reason for hiding this comment

Uh oh!

academo Sep 7, 2021

Choose a reason for hiding this comment

Uh oh!

dasansol92 Sep 7, 2021

Choose a reason for hiding this comment

Uh oh!

paul-tavares Sep 7, 2021

Choose a reason for hiding this comment

Uh oh!

joshdover left a comment

Choose a reason for hiding this comment

Uh oh!

paul-tavares Sep 7, 2021

Choose a reason for hiding this comment

Uh oh!

paul-tavares Sep 7, 2021

Choose a reason for hiding this comment

Uh oh!

paul-tavares left a comment

Choose a reason for hiding this comment

Uh oh!

kibanamachine commented Sep 7, 2021

💚 Build Succeeded

Metrics [docs]

History

Uh oh!

kibanamachine commented Sep 7, 2021

💚 Backport successful

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

dasansol92 commented Sep 6, 2021 •

edited

Loading