[Security Solution] Historical rules packages PoC by xcrzx · Pull Request #145851 · elastic/kibana

xcrzx · 2022-11-21T12:22:39Z

Summary

Related to: #137420

I've run a number of tests measuring the performance of the two versions of historical rules packages:

A package with a flat historical rules structure: a separate saved object for each historical rule version
A package with a composite historical rules structure: all historical versions of a given rule contained in a single saved object

On a relatively small total number of historical versions (< 10000 total versions, or 10 versions per rule for 1000 rules), the composite structure outperforms the flat one when installing the rules package:

Installation time/package size	5k versions	10k versions	17k versions	24k versions
Flat	5.27s / 18M	7.69 s / 36M	💥 / 59M	💥 / 79M
Composite	2.58s / 16M	2.97 s / 32M	4.11 s / 51M	4.22 s / 69M

The difference is visible but not so significant in those numbers. However, things become ugly for the flat stricture when we increase the total version number to 15-20k.

Maximum number of items in a nested field

The first problem that becomes visible is related to the maximum number of items in a nested field. It has already been discussed here and could be easily overcome by adding enabled: false to the mappings for the installed_kibana field:

kibana/x-pack/plugins/fleet/server/saved_objects/index.ts

Lines 261 to 267 in ab8dd04

    
           installed_kibana: { 
        
             type: 'nested', 
        
             properties: { 
        
               id: { type: 'keyword' }, 
        
               type: { type: 'keyword' }, 
        
             }, 
        
           },

Refresh ran out of slots and forced a refresh

After fixing the above error, the rules package becomes installable, but its installation starts to fail randomly, making Kibana unresponsive for some time. The problem seems to come from the Elasticsearch level. Console logs show dozens of warnings similar to this:

block until refresh ran out of slots and forced a refresh: [BulkShardRequest [[.kibana_8.7.0_001][0]] containing [delete {[.kibana_8.7.0][security-rule:d8fc1cca-93ed-43c1-bbb6-c0dd3eff2958:102.0.6]}] blocking until refresh]

During that time, all requests to Kibana fail with

{"statusCode":503,"error":"Service Unavailable","message":"connect EADDRNOTAVAIL 127.0.0.1:9200 - Local (0.0.0.0:0)"}

But after some time Elasticsearch cluster recovers by itself.

Created a ticket for the Fleet team: #147695

All shards failed

Another common error associated with the flat package installation is the following is shards failure. It occurs randomly and is not always easily reproduced:

{"statusCode":503,"error":"Service Unavailable","message":"all shards failed: search_phase_execution_exception: [no_shard_available_action_exception] Reason: null"}

After this failure, Elasticsearch doesn't recover by itself, and Kibana responds with:

{"statusCode":503,"error":"Service Unavailable","message":"[No shard available for [get [.kibana_8.7.0][space:default]: routing [null]]: no_shard_available_action_exception: [no_shard_available_action_exception] Reason: No shard available for [get [.kibana_8.7.0][space:default]: routing [null]]]: No shard available for [get [.kibana_8.7.0][space:default]: routing [null]]"}

Response timeout

Sometimes flat package installation just fails with a timeout.

{"statusCode":503,"error":"Service Unavailable","message":"Request timed out"}

Testing instructions

For reference: https://www.elastic.co/guide/en/integrations-developer/current/build-a-new-integration.html

Make sure elastic-package CLI and Docker are installed
Make sure the original prebuilt rules package is installed (visit any detection engine page to get it installed automatically).
If needed, generate more rule versions: curl -XPOST http://elastic:changeme@localhost:5601/kbn/internal/detection_engine/rules/prebuilt/_install_test_assets -d '{"num_versions_per_rule":10}' -H 'kbn-xsrf: true' -H 'Content-Type: application/json'
Build both flat and composite rules package variations:
1. cd fleet-packages/detection-rules-flat && elastic-package build --skip-validation
2. cd fleet-packages/detection-rules-composite && elastic-package build --skip-validation
From the Kibana root, start a copy of the package registry: elastic-package stack up --services package-registry
Copy the package registry certificate docker cp <container id>:/etc/ssl/package-registry/ca-cert.pem fleet-packages
Set registry URL in kibana.dev.yml: xpack.fleet.registryUrl: https://localhost:8080
Start Kibana specifying extra certificate NODE_EXTRA_CA_CERTS=./fleet-packages/ca-cert.pem yarn start
Go to http://localhost:5601/kbn/app/integrations/browse. You should find there two detection rules packages: Prebuilt detection rules (composite) and Prebuilt detection rules (flat)
To test package installation, use the following requests:
curl http://elastic:changeme@localhost:5601/kbn/api/fleet/epm/packages/security_rules_flat/8.3.2 -d '{"force":true}' -H 'kbn-xsrf: true' -H 'Content-Type: application/json'
curl http://elastic:changeme@localhost:5601/kbn/api/fleet/epm/packages/security_rules_composite/8.3.2 -d '{"force":true}' -H 'kbn-xsrf: true' -H 'Content-Type: application/json'

Conclusion

According to this PoC, the composite rule structure looks more stable. However, as outlined in another PoC, the flat structure provides more benefits when it comes to business logic implementation and overall looks more future-proof. My suggestion would be to fix the current performance issues that are associated with the flat structure and use it as a foundation for the rule customization work.

kibana-ci · 2022-12-27T15:26:56Z

💔 Build Failed

Failed CI Steps

Test Failures

[job] [logs] Jest Integration Tests #3 / checking migration metadata changes on all registered SO types detecting migration related changes in registered types
[job] [logs] Security Solution Tests #2 / Import rules Imports a custom rule with exceptions

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`securitySolution`	12.5MB	12.5MB	+6.0B

Saved Objects .kibana field count

Every field in each saved object type adds overhead to Elasticsearch. Kibana needs to keep the total field count below Elasticsearch's default limit of 1000 fields. Only specify field mappings for the fields you wish to search on or query. See https://www.elastic.co/guide/en/kibana/master/saved-objects-service.html#_mappings

id	before	after	diff
`epm-packages`	28	22	-6

Unknown metric groups

ESLint disabled in files

id	before	after	diff
`@kbn/analytics-shippers-fullstory`	1	2	+1
`@kbn/analytics-shippers-gainsight`	1	2	+1
`@kbn/apm-synthtrace`	3	4	+1
`@kbn/babel-register`	1	2	+1
`@kbn/bazel-packages`	1	2	+1
`@kbn/cli-dev-mode`	1	2	+1
`@kbn/core-deprecations-server-internal`	2	3	+1
`@kbn/core-metrics-collectors-server-internal`	2	4	+2
`@kbn/core-overlays-browser-internal`	2	4	+2
`@kbn/core-rendering-server-internal`	2	4	+2
`@kbn/core-saved-objects-api-server-internal`	1	2	+1
`@kbn/crypto-browser`	1	2	+1
`@kbn/docs-utils`	3	4	+1
`@kbn/es-query`	1	2	+1
`@kbn/eslint-plugin-disable`	39	76	+37
`@kbn/failed-test-reporter-cli`	1	2	+1
`@kbn/handlebars`	5	9	+4
`@kbn/i18n`	2	3	+1
`@kbn/monaco`	5	10	+5
`@kbn/optimizer`	2	4	+2
`@kbn/safer-lodash-set`	5	8	+3
`@kbn/securitysolution-io-ts-alerting-types`	9	18	+9
`@kbn/securitysolution-io-ts-list-types`	14	28	+14
`@kbn/shared-ux-file-picker`	1	2	+1
`@kbn/shared-ux-file-upload`	1	2	+1
`@kbn/shared-ux-link-redirect-app`	1	2	+1
`@kbn/storybook`	1	2	+1
`@kbn/ts-project-linter`	2	4	+2
`actions`	2	4	+2
`apm`	13	25	+12
`bfetch`	1	2	+1
`canvas`	4	7	+3
`cases`	17	27	+10
`cloudFullStory`	1	2	+1
`cloudSecurityPosture`	1	2	+1
`console`	3	5	+2
`crossClusterReplication`	2	4	+2
`dashboardEnhanced`	2	4	+2
`data`	2	4	+2
`dataViewEditor`	1	2	+1
`dataViews`	2	4	+2
`discover`	1	2	+1
`embeddable`	2	4	+2
`embeddableEnhanced`	2	4	+2
`enterpriseSearch`	1	2	+1
`eventAnnotation`	1	2	+1
`expressionError`	1	2	+1
`expressionImage`	1	2	+1
`expressionMetric`	1	2	+1
`expressionRepeatImage`	1	2	+1
`expressionRevealImage`	1	2	+1
`expressions`	5	9	+4
`expressionTagcloud`	1	2	+1
`files`	2	4	+2
`fleet`	9	15	+6
`home`	1	2	+1
`indexLifecycleManagement`	1	2	+1
`indexManagement`	4	8	+4
`infra`	8	15	+7
`inspector`	2	4	+2
`kibanaReact`	4	8	+4
`kibanaUsageCollection`	3	6	+3
`kibanaUtils`	2	4	+2
`lens`	3	6	+3
`lists`	5	10	+5
`maps`	28	42	+14
`ml`	3	6	+3
`monitoring`	7	14	+7
`observability`	7	10	+3
`osquery`	1	2	+1
`reporting`	1	2	+1
`ruleRegistry`	2	3	+1
`savedSearch`	1	2	+1
`screenshotting`	3	5	+2
`securitySolution`	75	134	+59
`share`	2	3	+1
`snapshotRestore`	1	2	+1
`synthetics`	6	11	+5
`telemetry`	6	11	+5
`transform`	3	6	+3
`triggersActionsUi`	3	5	+2
`uiActionsEnhanced`	4	8	+4
`urlForwarding`	1	2	+1
`usageCollection`	1	2	+1
`ux`	3	4	+1
`visTypeTimelion`	1	2	+1
`visTypeTimeseries`	3	6	+3
`visTypeVislib`	2	3	+1
`visTypeXy`	1	2	+1
`visualizations`	2	4	+2
total			+321

ESLint disabled line counts

id	before	after	diff
`@kbn/alerts`	2	3	+1
`@kbn/analytics`	1	2	+1
`@kbn/analytics-client`	18	36	+18
`@kbn/analytics-shippers-elastic-v3-browser`	4	8	+4
`@kbn/analytics-shippers-elastic-v3-server`	19	38	+19
`@kbn/analytics-shippers-fullstory`	5	10	+5
`@kbn/apm-config-loader`	2	4	+2
`@kbn/apm-synthtrace`	9	17	+8
`@kbn/cases-components`	1	2	+1
`@kbn/cli-dev-mode`	4	8	+4
`@kbn/coloring`	1	2	+1
`@kbn/content-management-table-list`	2	3	+1
`@kbn/core-analytics-browser-internal`	10	20	+10
`@kbn/core-application-browser-internal`	1	2	+1
`@kbn/core-chrome-browser-internal`	2	4	+2
`@kbn/core-elasticsearch-client-server-internal`	2	4	+2
`@kbn/core-elasticsearch-server-internal`	1	2	+1
`@kbn/core-fatal-errors-browser-internal`	4	8	+4
`@kbn/core-http-server-internal`	2	4	+2
`@kbn/core-i18n-browser-internal`	1	2	+1
`@kbn/core-logging-browser-internal`	3	6	+3
`@kbn/core-logging-server-internal`	8	16	+8
`@kbn/core-metrics-collectors-server-internal`	1	2	+1
`@kbn/core-plugins-server-internal`	4	8	+4
`@kbn/core-root-browser-internal`	4	8	+4
`@kbn/core-root-server-internal`	3	6	+3
`@kbn/core-saved-objects-api-server-internal`	6	12	+6
`@kbn/core-saved-objects-base-server-internal`	3	6	+3
`@kbn/core-ui-settings-server-internal`	1	2	+1
`@kbn/datemath`	1	2	+1
`@kbn/docs-utils`	8	16	+8
`@kbn/es`	1	2	+1
`@kbn/eslint-plugin-disable`	23	48	+25
`@kbn/eslint-plugin-imports`	1	2	+1
`@kbn/handlebars`	4	5	+1
`@kbn/i18n`	1	2	+1
`@kbn/i18n-react`	4	6	+2
`@kbn/interpreter`	5	8	+3
`@kbn/journeys`	1	2	+1
`@kbn/ml-string-hash`	2	4	+2
`@kbn/ml-url-state`	5	10	+5
`@kbn/monaco`	2	3	+1
`@kbn/optimizer`	4	7	+3
`@kbn/peggy-loader`	1	2	+1
`@kbn/rison`	2	4	+2
`@kbn/securitysolution-autocomplete`	1	2	+1
`@kbn/securitysolution-exception-list-components`	2	3	+1
`@kbn/securitysolution-io-ts-utils`	1	2	+1
`@kbn/securitysolution-list-api`	6	12	+6
`@kbn/securitysolution-list-hooks`	1	2	+1
`@kbn/securitysolution-t-grid`	1	2	+1
`@kbn/server-http-tools`	1	2	+1
`@kbn/shared-ux-avatar-user-profile-components`	2	4	+2
`@kbn/shared-ux-file-image`	2	4	+2
`@kbn/shared-ux-link-redirect-app`	1	2	+1
`@kbn/std`	1	2	+1
`@kbn/storybook`	1	2	+1
`@kbn/test`	6	12	+6
`@kbn/ui-shared-deps-npm`	1	2	+1
`@kbn/ui-shared-deps-src`	2	3	+1
`@kbn/ui-theme`	2	4	+2
`@kbn/user-profile-components`	3	6	+3
`actions`	22	42	+20
`advancedSettings`	2	3	+1
`aiops`	26	50	+24
`alerting`	70	121	+51
`apm`	75	144	+69
`canvas`	32	52	+20
`cases`	58	97	+39
`cloudChat`	2	4	+2
`cloudExperiments`	4	8	+4
`cloudFullStory`	1	2	+1
`cloudGainsight`	1	2	+1
`cloudSecurityPosture`	12	17	+5
`console`	36	71	+35
`controls`	2	4	+2
`core`	7	14	+7
`crossClusterReplication`	7	14	+7
`customIntegrations`	11	20	+9
`dashboard`	6	10	+4
`data`	52	92	+40
`dataViewFieldEditor`	9	14	+5
`dataViews`	12	16	+4
`dataVisualizer`	43	86	+43
`discover`	40	81	+41
`embeddable`	7	14	+7
`embeddableEnhanced`	3	4	+1
`enterpriseSearch`	8	15	+7
`esUiShared`	12	24	+12
`eventLog`	4	7	+3
`expressionError`	5	7	+2
`expressionGauge`	2	3	+1
`expressionHeatmap`	2	3	+1
`expressionLegacyMetricVis`	5	8	+3
`expressionMetricVis`	2	4	+2
`expressionPartitionVis`	3	5	+2
`expressions`	16	25	+9
`expressionTagcloud`	2	3	+1
`expressionXY`	1	2	+1
`fieldFormats`	5	7	+2
`fileUpload`	4	7	+3
`fleet`	56	102	+46
`globalSearch`	3	4	+1
`globalSearchBar`	1	2	+1
`graph`	4	8	+4
`grokdebugger`	1	2	+1
`home`	10	17	+7
`indexLifecycleManagement`	1	2	+1
`indexManagement`	14	28	+14
`infra`	45	84	+39
`ingestPipelines`	15	30	+15
`interactiveSetup`	10	20	+10
`kibanaReact`	12	23	+11
`kibanaUsageCollection`	6	10	+4
`kibanaUtils`	12	23	+11
`kubernetesSecurity`	7	11	+4
`lens`	19	37	+18
`licenseApiGuard`	1	2	+1
`lists`	18	32	+14
`maps`	39	76	+37
`ml`	559	1114	+555
`monitoring`	18	36	+18
`notifications`	4	8	+4
`observability`	47	90	+43
`osquery`	106	199	+93
`painlessLab`	1	2	+1
`presentationUtil`	10	18	+8
`profiling`	9	15	+6
`remoteClusters`	2	3	+1
`reporting`	4	7	+3
`rollup`	5	10	+5
`ruleRegistry`	1	2	+1
`savedObjectsFinder`	1	2	+1
`savedObjectsTagging`	2	4	+2
`savedObjectsTaggingOss`	1	2	+1
`screenshotMode`	1	2	+1
`screenshotting`	6	12	+6
`security`	24	47	+23
`securitySolution`	432	762	+330
`sessionView`	6	9	+3
`share`	7	13	+6
`snapshotRestore`	3	6	+3
`spaces`	1	2	+1
`stackAlerts`	25	46	+21
`stackConnectors`	78	113	+35
`synthetics`	69	123	+54
`taskManager`	22	41	+19
`telemetry`	1	2	+1
`telemetryCollectionManager`	7	14	+7
`telemetryManagementSection`	1	2	+1
`threatIntelligence`	6	10	+4
`timelines`	23	30	+7
`transform`	27	54	+27
`triggersActionsUi`	137	228	+91
`uiActions`	1	2	+1
`uiActionsEnhanced`	9	18	+9
`unifiedFieldList`	17	28	+11
`unifiedHistogram`	2	4	+2
`unifiedSearch`	26	48	+22
`upgradeAssistant`	9	17	+8
`urlDrilldown`	4	6	+2
`usageCollection`	2	3	+1
`ux`	9	16	+7
`visDefaultEditor`	4	7	+3
`visTypeTable`	4	5	+1
`visTypeTimelion`	5	7	+2
`visTypeTimeseries`	19	23	+4
`visTypeVega`	6	8	+2
`visTypeXy`	4	5	+1
`visualizations`	15	25	+10
`watcher`	3	6	+3
total			+2412

Total ESLint disabled count

id	before	after	diff
`@kbn/alerts`	2	3	+1
`@kbn/analytics`	1	2	+1
`@kbn/analytics-client`	18	36	+18
`@kbn/analytics-shippers-elastic-v3-browser`	4	8	+4
`@kbn/analytics-shippers-elastic-v3-server`	19	38	+19
`@kbn/analytics-shippers-fullstory`	6	12	+6
`@kbn/analytics-shippers-gainsight`	1	2	+1
`@kbn/apm-config-loader`	2	4	+2
`@kbn/apm-synthtrace`	12	21	+9
`@kbn/babel-register`	1	2	+1
`@kbn/bazel-packages`	1	2	+1
`@kbn/cases-components`	1	2	+1
`@kbn/cli-dev-mode`	5	10	+5
`@kbn/coloring`	1	2	+1
`@kbn/content-management-table-list`	2	3	+1
`@kbn/core-analytics-browser-internal`	10	20	+10
`@kbn/core-application-browser-internal`	1	2	+1
`@kbn/core-chrome-browser-internal`	2	4	+2
`@kbn/core-deprecations-server-internal`	2	3	+1
`@kbn/core-elasticsearch-client-server-internal`	2	4	+2
`@kbn/core-elasticsearch-server-internal`	1	2	+1
`@kbn/core-fatal-errors-browser-internal`	4	8	+4
`@kbn/core-http-server-internal`	2	4	+2
`@kbn/core-i18n-browser-internal`	1	2	+1
`@kbn/core-logging-browser-internal`	3	6	+3
`@kbn/core-logging-server-internal`	8	16	+8
`@kbn/core-metrics-collectors-server-internal`	3	6	+3
`@kbn/core-overlays-browser-internal`	2	4	+2
`@kbn/core-plugins-server-internal`	4	8	+4
`@kbn/core-rendering-server-internal`	2	4	+2
`@kbn/core-root-browser-internal`	4	8	+4
`@kbn/core-root-server-internal`	3	6	+3
`@kbn/core-saved-objects-api-server-internal`	7	14	+7
`@kbn/core-saved-objects-base-server-internal`	3	6	+3
`@kbn/core-ui-settings-server-internal`	1	2	+1
`@kbn/crypto-browser`	1	2	+1
`@kbn/datemath`	1	2	+1
`@kbn/docs-utils`	11	20	+9
`@kbn/es`	1	2	+1
`@kbn/es-query`	9	10	+1
`@kbn/eslint-plugin-disable`	62	124	+62
`@kbn/eslint-plugin-imports`	1	2	+1
`@kbn/failed-test-reporter-cli`	1	2	+1
`@kbn/handlebars`	9	14	+5
`@kbn/i18n`	3	5	+2
`@kbn/i18n-react`	4	6	+2
`@kbn/interpreter`	5	8	+3
`@kbn/journeys`	1	2	+1
`@kbn/ml-string-hash`	2	4	+2
`@kbn/ml-url-state`	5	10	+5
`@kbn/monaco`	7	13	+6
`@kbn/optimizer`	6	11	+5
`@kbn/peggy-loader`	1	2	+1
`@kbn/rison`	2	4	+2
`@kbn/safer-lodash-set`	5	8	+3
`@kbn/securitysolution-autocomplete`	1	2	+1
`@kbn/securitysolution-exception-list-components`	2	3	+1
`@kbn/securitysolution-io-ts-alerting-types`	9	18	+9
`@kbn/securitysolution-io-ts-list-types`	14	28	+14
`@kbn/securitysolution-io-ts-utils`	1	2	+1
`@kbn/securitysolution-list-api`	6	12	+6
`@kbn/securitysolution-list-hooks`	1	2	+1
`@kbn/securitysolution-t-grid`	1	2	+1
`@kbn/server-http-tools`	1	2	+1
`@kbn/shared-ux-avatar-user-profile-components`	2	4	+2
`@kbn/shared-ux-file-image`	2	4	+2
`@kbn/shared-ux-file-picker`	1	2	+1
`@kbn/shared-ux-file-upload`	1	2	+1
`@kbn/shared-ux-link-redirect-app`	2	4	+2
`@kbn/std`	1	2	+1
`@kbn/storybook`	2	4	+2
`@kbn/test`	6	12	+6
`@kbn/ts-project-linter`	2	4	+2
`@kbn/ui-shared-deps-npm`	1	2	+1
`@kbn/ui-shared-deps-src`	2	3	+1
`@kbn/ui-theme`	2	4	+2
`@kbn/user-profile-components`	3	6	+3
`actions`	24	46	+22
`advancedSettings`	2	3	+1
`aiops`	26	50	+24
`alerting`	71	122	+51
`apm`	88	169	+81
`bfetch`	6	7	+1
`canvas`	36	59	+23
`cases`	75	124	+49
`cloudChat`	2	4	+2
`cloudExperiments`	4	8	+4
`cloudFullStory`	2	4	+2
`cloudGainsight`	3	4	+1
`cloudSecurityPosture`	13	19	+6
`console`	39	76	+37
`controls`	2	4	+2
`core`	7	14	+7
`crossClusterReplication`	9	18	+9
`customIntegrations`	11	20	+9
`dashboard`	6	10	+4
`dashboardEnhanced`	7	9	+2
`data`	54	96	+42
`dataViewEditor`	3	4	+1
`dataViewFieldEditor`	9	14	+5
`dataViews`	14	20	+6
`dataVisualizer`	43	86	+43
`discover`	41	83	+42
`embeddable`	9	18	+9
`embeddableEnhanced`	5	8	+3
`enterpriseSearch`	9	17	+8
`esUiShared`	12	24	+12
`eventAnnotation`	1	2	+1
`eventLog`	4	7	+3
`expressionError`	6	9	+3
`expressionGauge`	2	3	+1
`expressionHeatmap`	2	3	+1
`expressionImage`	2	3	+1
`expressionLegacyMetricVis`	5	8	+3
`expressionMetric`	2	3	+1
`expressionMetricVis`	2	4	+2
`expressionPartitionVis`	3	5	+2
`expressionRepeatImage`	2	3	+1
`expressionRevealImage`	2	3	+1
`expressions`	21	34	+13
`expressionTagcloud`	3	5	+2
`expressionXY`	1	2	+1
`fieldFormats`	5	7	+2
`files`	3	5	+2
`fileUpload`	4	7	+3
`fleet`	65	117	+52
`globalSearch`	3	4	+1
`globalSearchBar`	1	2	+1
`graph`	4	8	+4
`grokdebugger`	1	2	+1
`home`	11	19	+8
`indexLifecycleManagement`	2	4	+2
`indexManagement`	18	36	+18
`infra`	53	99	+46
`ingestPipelines`	15	30	+15
`inspector`	3	5	+2
`interactiveSetup`	10	20	+10
`kibanaReact`	16	31	+15
`kibanaUsageCollection`	9	16	+7
`kibanaUtils`	14	27	+13
`kubernetesSecurity`	7	11	+4
`lens`	22	43	+21
`licenseApiGuard`	1	2	+1
`lists`	23	42	+19
`maps`	67	118	+51
`ml`	562	1120	+558
`monitoring`	25	50	+25
`notifications`	4	8	+4
`observability`	54	100	+46
`osquery`	107	201	+94
`painlessLab`	1	2	+1
`presentationUtil`	10	18	+8
`profiling`	9	15	+6
`remoteClusters`	2	3	+1
`reporting`	5	9	+4
`rollup`	5	10	+5
`ruleRegistry`	3	5	+2
`savedObjectsFinder`	1	2	+1
`savedObjectsTagging`	2	4	+2
`savedObjectsTaggingOss`	1	2	+1
`savedSearch`	1	2	+1
`screenshotMode`	1	2	+1
`screenshotting`	9	17	+8
`security`	26	49	+23
`securitySolution`	507	896	+389
`sessionView`	6	9	+3
`share`	9	16	+7
`snapshotRestore`	4	8	+4
`spaces`	1	2	+1
`stackAlerts`	25	46	+21
`stackConnectors`	82	117	+35
`synthetics`	75	134	+59
`taskManager`	22	41	+19
`telemetry`	7	13	+6
`telemetryCollectionManager`	7	14	+7
`telemetryManagementSection`	1	2	+1
`threatIntelligence`	6	10	+4
`timelines`	23	30	+7
`transform`	30	60	+30
`triggersActionsUi`	140	233	+93
`uiActions`	1	2	+1
`uiActionsEnhanced`	13	26	+13
`unifiedFieldList`	17	28	+11
`unifiedHistogram`	2	4	+2
`unifiedSearch`	26	48	+22
`upgradeAssistant`	9	17	+8
`urlDrilldown`	4	6	+2
`urlForwarding`	1	2	+1
`usageCollection`	3	5	+2
`ux`	12	20	+8
`visDefaultEditor`	5	8	+3
`visTypeTable`	4	5	+1
`visTypeTimelion`	6	9	+3
`visTypeTimeseries`	22	29	+7
`visTypeVega`	6	8	+2
`visTypeVislib`	3	4	+1
`visTypeXy`	5	7	+2
`visualizations`	17	29	+12
`watcher`	3	6	+3
total			+2733

History

💔 Build #95655 failed 17933ebbb222356a7e02141b035939805b801fde
💔 Build #94365 failed aaef9a1411a27d2614a3ff49b0a853c320321569
💔 Build #94299 failed 46b2732673298cab4d838d6249f8656eb1672421
💔 Build #91024 failed b43fad9552b9ac1a3d11110e3934cbd89174f496

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @xcrzx

…ects (#148141) **Resolves: #147695, #148174 **Related to: #145851, #137420 ## Summary This PR improves the stability of the Fleet packages installation process with many saved objects. 1. Changed mappings of the `installed_kibana` and `package_assets` fields from `nested` to `object` with `enabled: false`. Values of those fields were retrieved from `_source`, and no queries or aggregations were performed against them. So the mappings were unused, while during the installation of packages containing more than 10,000 saved objects, an error was thrown due to the nested field limitations: ``` Error installing security_detection_engine 8.4.1: The number of nested documents has exceeded the allowed limit of [10000]. This limit can be set by changing the [index.mapping.nested_objects.limit] index level setting. ``` 2. Improved the deletion of previous package assets by switching from sending multiple `savedObjectsClient.delete` requests in parallel to a single `savedObjectsClient.bulkDelete` request. Multiple parallel requests were causing the Elasticsearch cluster to stop responding for some time; see [this ticket](#147695) for more info. **Before** ![Screenshot 2022-12-28 at 11 09 35](https://user-images.githubusercontent.com/1938181/209816219-ade6dd0a-0d56-4acc-929e-b88571f0fe81.png) **After** ![Screenshot 2022-12-28 at 13 56 44](https://user-images.githubusercontent.com/1938181/209816209-16c69922-4ae2-4589-9aa4-5a28050037f4.png)

xcrzx · 2023-01-04T13:50:59Z

Closing this PR as both data structures were thoroughly tested.

xcrzx added Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Detection Rule Management Security Detection Rule Management Team labels Nov 21, 2022

xcrzx self-assigned this Nov 21, 2022

banderror added the Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules area label Dec 12, 2022

xcrzx mentioned this pull request Dec 16, 2022

[Fleet] Packages with a large number of saved objects in them cause Kibana to crash #147695

Closed

Historical rules packages PoC

b6cc24a

xcrzx mentioned this pull request Dec 28, 2022

[Fleet] Fix crashes when installing a package with a lot of saved objects #148141

Merged

xcrzx closed this Jan 4, 2023

xcrzx mentioned this pull request Jan 4, 2023

[Security Solution] PoC of Prebuilt Detection Rules package with historical versions #137420

Closed

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution] Historical rules packages PoC#145851

[Security Solution] Historical rules packages PoC#145851
xcrzx wants to merge 1 commit intoelastic:mainfrom
xcrzx:rules-package-poc

xcrzx commented Nov 21, 2022 •

edited

Loading

Uh oh!

kibana-ci commented Dec 27, 2022 •

edited

Loading

ESLint disabled in files

ESLint disabled line counts

Total ESLint disabled count

Uh oh!

xcrzx commented Jan 4, 2023 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	installed_kibana: {
	type: 'nested',
	properties: {
	id: { type: 'keyword' },
	type: { type: 'keyword' },
	},
	},

Conversation

xcrzx commented Nov 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Maximum number of items in a nested field

Refresh ran out of slots and forced a refresh

All shards failed

Response timeout

Testing instructions

Conclusion

Uh oh!

kibana-ci commented Dec 27, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

Test Failures

Metrics [docs]

Async chunks

Saved Objects .kibana field count

ESLint disabled in files

ESLint disabled line counts

Total ESLint disabled count

History

Uh oh!

xcrzx commented Jan 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

xcrzx commented Nov 21, 2022 •

edited

Loading

kibana-ci commented Dec 27, 2022 •

edited

Loading

xcrzx commented Jan 4, 2023 •

edited

Loading