Skip to content

[Fleet] Install security_rule assets as saved objects#95885

Merged
rw-access merged 18 commits intoelastic:masterfrom
rw-access:fleet-install-security-rule-asset
Apr 8, 2021
Merged

[Fleet] Install security_rule assets as saved objects#95885
rw-access merged 18 commits intoelastic:masterfrom
rw-access:fleet-install-security-rule-asset

Conversation

@rw-access
Copy link
Copy Markdown
Contributor

@rw-access rw-access commented Mar 31, 2021

Summary

This adds a new security_rule asset type per elastic/package-spec#142.
I saw #94950 and copied the approach for the integration test that I saw there. No asset UUIDs yet though.

Example integration was published to elastic/integrations#797 and elastic/package-storage#1082.

Checklist

Delete any items that are not applicable to this PR.

For maintainers

@rw-access rw-access mentioned this pull request Mar 31, 2021
Merged
@rw-access
Copy link
Copy Markdown
Contributor Author

rw-access commented Mar 31, 2021 •
edited
Loading

@jgowdyelastic if you don't mind helping me out, I ran into this error. Did you see this too or have any guesses? Code-wise everything looks the same AFAICT

Details 
           └-> "before each" hook: beforeSetupWithDockerRegistry for "should have installed the kibana assets"
           │ debg Gettings saved object: {"type":"index-pattern","id":"logs-*"}
           │ debg Gettings saved object: {"type":"index-pattern","id":"metrics-*"}
           │ debg Gettings saved object: {"type":"dashboard","id":"sample_dashboard"}
           │ debg Gettings saved object: {"type":"dashboard","id":"sample_dashboard2"}
           │ debg Gettings saved object: {"type":"visualization","id":"sample_visualization"}
           │ debg Gettings saved object: {"type":"search","id":"sample_search"}
           │ debg Gettings saved object: {"type":"lens","id":"sample_lens"}
           │ debg Gettings saved object: {"type":"ml-module","id":"sample_ml_module"}
           │ debg Gettings saved object: {"type":"security-rule","id":"sample_security_rule"}
           └- ✖ fail: Fleet Endpoints EPM Endpoints installs and uninstalls all assets installs all assets when installing a package for the first time should have installed the kibana assets
           │      Error: Request failed with status code 404
           │       at createError (node_modules/axios/lib/core/createError.js:16:15)
           │       at settle (node_modules/axios/lib/core/settle.js:17:12)
           │       at IncomingMessage.handleStreamEnd (node_modules/axios/lib/adapters/http.js:260:11)
           │       at endReadableNT (internal/streams/readable.js:1327:12)
           │       at processTicksAndRejections (internal/process/task_queues.js:80:21)
           │ 
           │ 
         └-> should create an index pattern with the package fields
           └-> "before each" hook: global before each for "should create an index pattern with the package fields"
           └-> "before each" hook: beforeSetupWithDockerRegistry for "should create an index pattern with the package fields"
           │ debg Gettings saved object: {"type":"index-pattern","id":"logs-*"}
           │ debg Gettings saved object: {"type":"index-pattern","id":"metrics-*"}
           └- ✓ pass  (67ms) "Fleet Endpoints EPM Endpoints installs and uninstalls all assets installs all assets when installing a package for the first time should create an index pattern with the package fields"
         └-> should have created the correct saved object
           └-> "before each" hook: global before each for "should have created the correct saved object"
           └-> "before each" hook: beforeSetupWithDockerRegistry for "should have created the correct saved object"
           │ debg Gettings saved object: {"type":"epm-packages","id":"all_assets"}
           └- ✖ fail: Fleet Endpoints EPM Endpoints installs and uninstalls all assets installs all assets when installing a package for the first time should have created the correct saved object
           │       Error: expected { installed_kibana: 
           │    [ { id: 'sample_dashboard', type: 'dashboard' },
           │      { id: 'sample_dashboard2', type: 'dashboard' },
           │      { id: 'test-*', type: 'index-pattern' },
           │      { id: 'sample_lens', type: 'lens' },
           │      { id: 'sample_ml_module', type: 'ml-module' },
           │      { id: 'sample_search', type: 'search' },
           │      { id: 'sample_visualization', type: 'visualization' } ],
           │   installed_es: 
           │    [ { id: 'logs-all_assets.test_logs-all_assets',
           │        type: 'data_stream_ilm_policy' },
           │      { id: 'metrics-all_assets.test_metrics-all_assets',
           │        type: 'data_stream_ilm_policy' },
           │      { id: 'logs-all_assets.test_logs',
           │        type: 'index_template' },
           │      { id: 'metrics-all_assets.test_metrics',
           │        type: 'index_template' },
           │      { id: 'logs-all_assets.test_logs-0.1.0',
           │        type: 'ingest_pipeline' },
           │      { id: 'logs-all_assets.test_logs-0.1.0-pipeline1',
           │        type: 'ingest_pipeline' },
           │      { id: 'logs-all_assets.test_logs-0.1.0-pipeline2',
           │        type: 'ingest_pipeline' },
           │      { id: 'all_assets.test-default-0.1.0',
           │        type: 'transform' } ],
           │   package_assets: 
           │    [ { id: '333a22a1-e639-5af5-ae62-907ffc83d603',
           │        type: 'epm-packages-assets' },
           │      { id: '256f3dad-6870-56c3-80a1-8dfa11e2d568',
           │        type: 'epm-packages-assets' },
           │      { id: '3fa0512f-bc01-5c2e-9df1-bc2f2a8259c8',
           │        type: 'epm-packages-assets' },
           │      { id: 'ea334ad8-80c2-5acd-934b-2a377290bf97',
           │        type: 'epm-packages-assets' },
           │      { id: '96c6eb85-fe2e-56c6-84be-5fda976796db',
           │        type: 'epm-packages-assets' },
           │      { id: '2d73a161-fa69-52d0-aa09-1bdc691b95bb',
           │        type: 'epm-packages-assets' },
           │      { id: '0a00c2d2-ce63-5b9c-9aa0-0cf1938f7362',
           │        type: 'epm-packages-assets' },
           │      { id: '691f0505-18c5-57a6-9f40-06e8affbdf7a',
           │        type: 'epm-packages-assets' },
           │      { id: 'b36e6dd0-58f7-5dd0-a286-8187e4019274',
           │        type: 'epm-packages-assets' },
           │      { id: 'f839c76e-d194-555a-90a1-3265a45789e4',
           │        type: 'epm-packages-assets' },
           │      { id: '9af7bbb3-7d8a-50fa-acc9-9dde6f5efca2',
           │        type: 'epm-packages-assets' },
           │      { id: '1e97a20f-9d1c-529b-8ff2-da4e8ba8bb71',
           │        type: 'epm-packages-assets' },
           │      { id: '8cfe0a2b-7016-5522-87e4-6d352360d1fc',
           │        type: 'epm-packages-assets' },
           │      { id: 'bd5ff3c5-655e-5385-9918-b60ff3040aad',
           │        type: 'epm-packages-assets' },
           │      { id: '0954ce3b-3165-5c1f-a4c0-56eb5f2fa487',
           │        type: 'epm-packages-assets' },
           │      { id: '60d6d054-57e4-590f-a580-52bf3f5e7cca',
           │        type: 'epm-packages-assets' },
           │      { id: '47758dc2-979d-5fbe-a2bd-9eded68a5a43',
           │        type: 'epm-packages-assets' },
           │      { id: '318959c9-997b-5a14-b328-9fc7355b4b74',
           │        type: 'epm-packages-assets' },
           │      { id: 'e21b59b5-eb76-5ab0-bef2-1c8e379e6197',
           │        type: 'epm-packages-assets' },
           │      { id: '4c758d70-ecf1-56b3-b704-6d8374841b34',
           │        type: 'epm-packages-assets' },
           │      { id: 'e786cbd9-0f3b-5a0b-82a6-db25145ebf58',
           │        type: 'epm-packages-assets' },
           │      { id: 'd8b175c3-0d42-5ec7-90c1-d1e4b307a4c2',
           │        type: 'epm-packages-assets' },
           │      { id: '53c94591-aa33-591d-8200-cd524c2a0561',
           │        type: 'epm-packages-assets' },
           │      { id: 'b658d2d4-752e-54b8-afc2-4c76155c1466',
           │        type: 'epm-packages-assets' } ],
           │   es_index_patterns: 
           │    { test_logs: 'logs-all_assets.test_logs-*',
           │      test_metrics: 'metrics-all_assets.test_metrics-*' },
           │   name: 'all_assets',
           │   version: '0.1.0',
           │   internal: false,
           │   removable: true,
           │   install_version: '0.1.0',
           │   install_status: 'installed',
           │   install_started_at: '2021-03-31T18:44:09.506Z',
           │   install_source: 'registry' } to sort of equal { installed_kibana: 
           │    [ { id: 'sample_dashboard', type: 'dashboard' },
           │      { id: 'sample_dashboard2', type: 'dashboard' },
           │      { id: 'test-*', type: 'index-pattern' },
           │      { id: 'sample_lens', type: 'lens' },
           │      { id: 'sample_ml_module', type: 'ml-module' },
           │      { id: 'sample_search', type: 'search' },
           │      { id: 'sample_security_rule', type: 'security-rule' },
           │      { id: 'sample_visualization', type: 'visualization' } ],
           │   installed_es: 
           │    [ { id: 'logs-all_assets.test_logs-all_assets',
           │        type: 'data_stream_ilm_policy' },
           │      { id: 'metrics-all_assets.test_metrics-all_assets',
           │        type: 'data_stream_ilm_policy' },
           │      { id: 'logs-all_assets.test_logs',
           │        type: 'index_template' },
           │      { id: 'metrics-all_assets.test_metrics',
           │        type: 'index_template' },
           │      { id: 'logs-all_assets.test_logs-0.1.0',
           │        type: 'ingest_pipeline' },
           │      { id: 'logs-all_assets.test_logs-0.1.0-pipeline1',
           │        type: 'ingest_pipeline' },
           │      { id: 'logs-all_assets.test_logs-0.1.0-pipeline2',
           │        type: 'ingest_pipeline' },
           │      { id: 'all_assets.test-default-0.1.0',
           │        type: 'transform' } ],
           │   es_index_patterns: 
           │    { test_logs: 'logs-all_assets.test_logs-*',
           │      test_metrics: 'metrics-all_assets.test_metrics-*' },
           │   package_assets: 
           │    [ { id: '333a22a1-e639-5af5-ae62-907ffc83d603',
           │        type: 'epm-packages-assets' },
           │      { id: '256f3dad-6870-56c3-80a1-8dfa11e2d568',
           │        type: 'epm-packages-assets' },
           │      { id: '3fa0512f-bc01-5c2e-9df1-bc2f2a8259c8',
           │        type: 'epm-packages-assets' },
           │      { id: 'ea334ad8-80c2-5acd-934b-2a377290bf97',
           │        type: 'epm-packages-assets' },
           │      { id: '96c6eb85-fe2e-56c6-84be-5fda976796db',
           │        type: 'epm-packages-assets' },
           │      { id: '2d73a161-fa69-52d0-aa09-1bdc691b95bb',
           │        type: 'epm-packages-assets' },
           │      { id: '0a00c2d2-ce63-5b9c-9aa0-0cf1938f7362',
           │        type: 'epm-packages-assets' },
           │      { id: '691f0505-18c5-57a6-9f40-06e8affbdf7a',
           │        type: 'epm-packages-assets' },
           │      { id: 'b36e6dd0-58f7-5dd0-a286-8187e4019274',
           │        type: 'epm-packages-assets' },
           │      { id: 'f839c76e-d194-555a-90a1-3265a45789e4',
           │        type: 'epm-packages-assets' },
           │      { id: '9af7bbb3-7d8a-50fa-acc9-9dde6f5efca2',
           │        type: 'epm-packages-assets' },
           │      { id: '1e97a20f-9d1c-529b-8ff2-da4e8ba8bb71',
           │        type: 'epm-packages-assets' },
           │      { id: '8cfe0a2b-7016-5522-87e4-6d352360d1fc',
           │        type: 'epm-packages-assets' },
           │      { id: 'bd5ff3c5-655e-5385-9918-b60ff3040aad',
           │        type: 'epm-packages-assets' },
           │      { id: '0954ce3b-3165-5c1f-a4c0-56eb5f2fa487',
           │        type: 'epm-packages-assets' },
           │      { id: '60d6d054-57e4-590f-a580-52bf3f5e7cca',
           │        type: 'epm-packages-assets' },
           │      { id: '47758dc2-979d-5fbe-a2bd-9eded68a5a43',
           │        type: 'epm-packages-assets' },
           │      { id: '318959c9-997b-5a14-b328-9fc7355b4b74',
           │        type: 'epm-packages-assets' },
           │      { id: 'e21b59b5-eb76-5ab0-bef2-1c8e379e6197',
           │        type: 'epm-packages-assets' },
           │      { id: '4c758d70-ecf1-56b3-b704-6d8374841b34',
           │        type: 'epm-packages-assets' },
           │      { id: 'e786cbd9-0f3b-5a0b-82a6-db25145ebf58',
           │        type: 'epm-packages-assets' },
           │      { id: 'd8b175c3-0d42-5ec7-90c1-d1e4b307a4c2',
           │        type: 'epm-packages-assets' },
           │      { id: '53c94591-aa33-591d-8200-cd524c2a0561',
           │        type: 'epm-packages-assets' },
           │      { id: 'b658d2d4-752e-54b8-afc2-4c76155c1466',
           │        type: 'epm-packages-assets' } ],
           │   name: 'all_assets',
           │   version: '0.1.0',
           │   internal: false,
           │   removable: true,
           │   install_version: '0.1.0',
           │   install_status: 'installed',
           │   install_started_at: '2021-03-31T18:44:09.506Z',
           │   install_source: 'registry' }
           │       + expected - actual
           │ 
           │              "id": "sample_search"
           │              "type": "search"
           │            }
           │            {
           │       +      "id": "sample_security_rule"
           │       +      "type": "security-rule"
           │       +    }
           │       +    {
           │              "id": "sample_visualization"
           │              "type": "visualization"
           │            }
           │          ]
           │       
           │       at Assertion.assert (node_modules/@kbn/expect/expect.js:100:11)
           │       at Assertion.eql (node_modules/@kbn/expect/expect.js:244:8)
           │       at Context.<anonymous> (x-pack/test/fleet_api_integration/apis/epm/install_remove_assets.ts:454:23)
           │       at processTicksAndRejections (internal/process/task_queues.js:93:5)
           │       at Object.apply (node_modules/@kbn/test/src/functional_test_runner/lib/mocha/wrap_function.js:73:16)
           │ 
           │ 
         └-> "after all" hook for "should have created the correct saved object"
         └-> "after all" hook for "should have created the correct saved object"
       └-: uninstalls all assets when uninstalling a package

Got the same thing again for the 0.1.0 -> 0.2.0 upgrade script.

@rw-access rw-access mentioned this pull request Apr 1, 2021
Merged
3 tasks
@rw-access rw-access added Feature:Fleet Fleet team's agent central management project release_note:skip Skip the PR/issue when compiling release notes labels Apr 1, 2021
@rw-access rw-access marked this pull request as ready for review April 1, 2021 16:09
@rw-access rw-access requested a review from a team as a code owner April 1, 2021 16:09
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Apr 1, 2021

Pinging @elastic/fleet (Feature:Fleet)

@rw-access
Copy link
Copy Markdown
Contributor Author

rw-access commented Apr 1, 2021

@elasticmachine update branch

@nchaulet nchaulet self-requested a review April 1, 2021 18:04
@jen-huang jen-huang added the Team:Fleet Team label for Observability Data Collection Fleet team label Apr 1, 2021
nchaulet
nchaulet approved these changes Apr 1, 2021
View reviewed changes
Copy link
Copy Markdown
Member

@nchaulet nchaulet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me 🚀

@rw-access rw-access added the auto-backport Deprecated - use backport:version if exact versions are needed label Apr 1, 2021
@rw-access rw-access requested a review from a team as a code owner April 2, 2021 15:22
rw-access
rw-access commented Apr 2, 2021
View reviewed changes
x-pack/plugins/security_solution/server/lib/detection_engine/rules/saved_object_mappings.ts
mappings: ruleStatusSavedObjectMappings,
};

export const ruleAssetSavedObjectType = 'security-rule';
Copy link
Copy Markdown
Contributor Author

@rw-access rw-access Apr 2, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@FrankHassanabad is this change okay?
for some reason, it doesn't look like fleet hasn't yet recognized this saved object when running tests (at least locally)

Copy link
Copy Markdown
Contributor

@FrankHassanabad FrankHassanabad Apr 8, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be ok. Don't see anything weird here.

@rw-access rw-access mentioned this pull request Apr 2, 2021
Merged
ruflin
ruflin reviewed Apr 4, 2021
View reviewed changes
x-pack/plugins/fleet/public/applications/fleet/sections/epm/constants.tsx Outdated
map: 'Map',
data_stream_ilm_policy: 'Data Stream ILM Policy',
lens: 'Lens',
security_rule: 'Detection Engine Rule',
Copy link
Copy Markdown
Contributor

@ruflin ruflin Apr 4, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I stumble over the mismatch of naming and description. Is this something to worry about?

Copy link
Copy Markdown
Contributor Author

@rw-access rw-access Apr 6, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated to "Security Rule", confirming with PM on the naming

@jasonrhodes jasonrhodes mentioned this pull request Apr 7, 2021
Closed
FrankHassanabad
FrankHassanabad approved these changes Apr 8, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor

@FrankHassanabad FrankHassanabad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks

spong
spong approved these changes Apr 8, 2021
View reviewed changes
Copy link
Copy Markdown
Member

@spong spong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked out, tested locally, and Security Solution changes LGTM! So awesome seeing Detection Rules under Fleet now, great stuff @randomuserid! 🙂🎉🚀🙌

image

@spong
Copy link
Copy Markdown
Member

spong commented Apr 8, 2021

@elasticmachine merge upstream

@rw-access rw-access enabled auto-merge (squash) April 8, 2021 03:49
@kibanamachine
Copy link
Copy Markdown
Contributor

kibanamachine commented Apr 8, 2021

💚 Build Succeeded

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
fleet 710.0KB 710.0KB +30.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
fleet 345.3KB 345.4KB +96.0B

Saved Objects .kibana field count

Every field in each saved object type adds overhead to Elasticsearch. Kibana needs to keep the total field count below Elasticsearch's default limit of 1000 fields. Only specify field mappings for the fields you wish to search on or query. See https://www.elastic.co/guide/en/kibana/master/development-plugin-saved-objects.html#_mappings

id before after diff
security-rule - 4 +4

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@rw-access rw-access merged commit 9396534 into elastic:master Apr 8, 2021
@kibanamachine kibanamachine mentioned this pull request Apr 8, 2021
Merged
@kibanamachine
Copy link
Copy Markdown
Contributor

kibanamachine commented Apr 8, 2021

💚 Backport successful

✅ 7.x / #96527

This backport PR will be merged automatically after passing CI.

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Apr 8, 2021
@rw-access @kibanamachine
[Fleet] Install security_rule assets as saved objects (elastic#95885)
85c1e88 
* [Fleet] Install security_rule assets as saved objects

* Add security-rule to update_assets.ts

* Update UUIDs for security_rule asset

* Change .type to match the saved object type not the asset type

* Add saved object mapping for security-rule

* Make SO non-hidden

* Fix SO mapping for security-rule

* Make security-rule a non-hidden asset
kibanamachine added a commit that referenced this pull request Apr 8, 2021
@kibanamachine @rw-access
[Fleet] Install security_rule assets as saved objects (#95885) (#96527)
95992f0 
* [Fleet] Install security_rule assets as saved objects

* Add security-rule to update_assets.ts

* Update UUIDs for security_rule asset

* Change .type to match the saved object type not the asset type

* Add saved object mapping for security-rule

* Make SO non-hidden

* Fix SO mapping for security-rule

* Make security-rule a non-hidden asset

Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ruflin ruflin ruflin left review comments

@nchaulet nchaulet nchaulet approved these changes

@spong spong spong approved these changes

@jgowdyelastic jgowdyelastic Awaiting requested review from jgowdyelastic

+2 more reviewers

@skh skh skh left review comments

@FrankHassanabad FrankHassanabad FrankHassanabad approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

auto-backport Deprecated - use backport:version if exact versions are needed Feature:Fleet Fleet team's agent central management project release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team v7.13.0 v8.0.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

9 participants

@rw-access @elasticmachine @spong @kibanamachine @skh @ruflin @FrankHassanabad @nchaulet @jen-huang