Skip to content

[Fleet] Fix crashes when installing a package with a lot of saved objects#148141

Merged
xcrzx merged 1 commit intoelastic:mainfrom
xcrzx:fleet-bulk-delete
Jan 3, 2023
Merged

[Fleet] Fix crashes when installing a package with a lot of saved objects#148141
xcrzx merged 1 commit intoelastic:mainfrom
xcrzx:fleet-bulk-delete

Conversation

@xcrzx
Copy link
Copy Markdown
Contributor

@xcrzx xcrzx commented Dec 28, 2022 •
edited
Loading

Resolves: #147695, #148174
Related to: #145851, #137420

Summary

This PR improves the stability of the Fleet packages installation process with many saved objects.

  1. Changed mappings of the installed_kibana and package_assets fields from nested to object with enabled: false. Values of those fields were retrieved from _source, and no queries or aggregations were performed against them. So the mappings were unused, while during the installation of packages containing more than 10,000 saved objects, an error was thrown due to the nested field limitations:

    Error installing security_detection_engine 8.4.1: The number of nested documents has exceeded the allowed limit of 
[10000].
This limit can be set by changing the [index.mapping.nested_objects.limit] index level setting.

  2. Improved the deletion of previous package assets by switching from sending multiple savedObjectsClient.delete requests in parallel to a single savedObjectsClient.bulkDelete request. Multiple parallel requests were causing the Elasticsearch cluster to stop responding for some time; see this ticket for more info.

Before
Screenshot 2022-12-28 at 11 09 35

After
Screenshot 2022-12-28 at 13 56 44

@xcrzx xcrzx self-assigned this Dec 28, 2022
@xcrzx xcrzx added v8.7.0 Team:Fleet Team label for Observability Data Collection Fleet team release_note:skip Skip the PR/issue when compiling release notes performance labels Dec 28, 2022
@xcrzx xcrzx marked this pull request as ready for review December 28, 2022 13:04
@xcrzx xcrzx requested review from a team as code owners December 28, 2022 13:04
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Dec 28, 2022

Pinging @elastic/fleet (Team:Fleet)

@xcrzx xcrzx changed the title Use SavedObjectsClient.bulkDelete to delete package assets [Fleet] Fix crashes when installing a package with a lot of saved objects Dec 28, 2022
kpollich
kpollich approved these changes Dec 28, 2022
View reviewed changes
Copy link
Copy Markdown
Member

@kpollich kpollich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code changes look small in scope, but with big impact. Really appreciate the improvements here, and the before/after traces you provided. LGTM 🚀

@banderror banderror linked an issue Dec 29, 2022 that may be closed by this pull request
[Fleet] Remove the mapping for the nested istalled_kibana field #148174
Closed
@kibana-ci
Copy link
Copy Markdown

kibana-ci commented Jan 3, 2023 •
edited
Loading

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

  • [job] [logs] Jest Tests #7 / AllCasesListGeneric Actions Row actions should disable row actions when bulk selecting all cases
  • [job] [logs] Jest Tests #7 / AllCasesListGeneric Actions Row actions should disable row actions when selecting a case
  • [job] [logs] Jest Tests #7 / AllCasesListGeneric Actions Row actions update the status of a case: critical

Metrics [docs]

Saved Objects .kibana field count

Every field in each saved object type adds overhead to Elasticsearch. Kibana needs to keep the total field count below Elasticsearch's default limit of 1000 fields. Only specify field mappings for the fields you wish to search on or query. See https://www.elastic.co/guide/en/kibana/master/saved-objects-service.html#_mappings

id before after diff
epm-packages 28 22 -6

History

  • 💚 Build #97227 succeeded 5d3d24b0644c9d2e1e158edfbe8d14e2e2ac89b7
  • 💔 Build #97207 failed 36cc2bfb528045785f17a54827a2110ebef0c3ce

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @xcrzx

@xcrzx xcrzx merged commit ae60734 into elastic:main Jan 3, 2023
@kibanamachine kibanamachine added the backport:skip This PR does not require backporting label Jan 3, 2023
@xcrzx xcrzx deleted the fleet-bulk-delete branch January 3, 2023 11:07
@juliaElastic juliaElastic mentioned this pull request Jan 5, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rudolf rudolf rudolf approved these changes

@kpollich kpollich kpollich approved these changes

Assignees

@xcrzx xcrzx

Labels

backport:skip This PR does not require backporting performance release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team v8.7.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Fleet] Remove the mapping for the nested istalled_kibana field [Fleet] Packages with a large number of saved objects in them cause Kibana to crash

6 participants

@xcrzx @elasticmachine @kibana-ci @rudolf @kpollich @kibanamachine