[SIEM][Detection Engine][Lists] Adds conflict versioning and io-ts improvements to lists#72337
Merged
FrankHassanabad merged 11 commits intoelastic:masterfrom Jul 20, 2020
Merged
Conversation
Contributor
|
Pinging @elastic/siem (Team:SIEM) |
Contributor
|
@FrankHassanabad Pulled this down, ran through several tests using the scripts. Updates to lists, list items, exception lists, and exception list items work with and without |
madirey
approved these changes
Jul 19, 2020
Contributor
💚 Build SucceededBuild metricspage load bundle size
History
To update your PR or re-run it, just comment with: |
FrankHassanabad
added a commit
to FrankHassanabad/kibana
that referenced
this pull request
Jul 20, 2020
…provements to lists (elastic#72337) ## Summary * Adds conflict versioning by exposing the "_version" from the saved object system. It renames "version" to "_version" so that we can use regular "version" later for versioning things for pre-packaged lists abilities. * Utilizes `t.OutputOf` in the requests and the data types to give us more correctly types * Removes the `Identity` utility as that is adding confusion and can confuse vs code rather than improves things * Removes extra types that were causing confusion which was an idiom from io-ts * Changes the wording of `Partial` by removing that and instead focuses the request types on either client side or server side at this point. NOTE: The UI can migrate to holding onto the `_version` and then push it back down when it wants to migrate to using the conflict resolution. If the UI does not push it down, then a value of undefined will be used which is indicating that no conflict errors are wanted. Output example of posting an exception list: ❯ ./post_exception_list.sh ```ts { "_tags": [ "endpoint", "process", "malware", "os:linux" ], "_version": "Wzk4NiwxXQ==", "created_at": "2020-07-17T18:59:22.872Z", "created_by": "yo", "description": "This is a sample endpoint type exception", "id": "a08795b0-c85f-11ea-b1a6-c155df988a92", "list_id": "simple_list", "name": "Sample Endpoint Exception List", "namespace_type": "single", "tags": [ "user added string for a tag", "malware" ], "tie_breaker_id": "b789ec05-3e0f-4344-a156-0c0f5b6e2f9c", "type": "detection", "updated_at": "2020-07-17T18:59:22.891Z", "updated_by": "yo" } ``` Output example of posting an exception list item ❯ ./post_exception_list_item.sh ```ts { "_tags": [ "endpoint", "process", "malware", "os:linux" ], "_version": "Wzk4NywxXQ==", "comments": [], "created_at": "2020-07-17T18:59:30.286Z", "created_by": "yo", "description": "This is a sample endpoint type exception", "entries": [ { "field": "actingProcess.file.signer", "operator": "excluded", "type": "exists" }, { "field": "host.name", "operator": "included", "type": "match_any", "value": [ "some host", "another host" ] } ], "id": "a4f2b800-c85f-11ea-b1a6-c155df988a92", "item_id": "simple_list_item", "list_id": "simple_list", "name": "Sample Endpoint Exception List", "namespace_type": "single", "tags": [ "user added string for a tag", "malware" ], "tie_breaker_id": "1dc456bc-7aa9-44b4-bca3-131689cf729f", "type": "simple", "updated_at": "2020-07-17T18:59:30.304Z", "updated_by": "yo" } ``` Output example of when you get an exception list: ❯ ./get_exception_list.sh simple_list ```ts { "_tags": [ "endpoint", "process", "malware", "os:linux" ], "_version": "WzEwNzcsMV0=", "created_at": "2020-07-17T18:59:22.872Z", "created_by": "yo", "description": "Different description", "id": "a08795b0-c85f-11ea-b1a6-c155df988a92", "list_id": "simple_list", "name": "Sample Endpoint Exception List", "namespace_type": "single", "tags": [ "user added string for a tag", "malware" ], "tie_breaker_id": "b789ec05-3e0f-4344-a156-0c0f5b6e2f9c", "type": "endpoint", "updated_at": "2020-07-17T20:01:24.958Z", "updated_by": "yo" } ``` Example of the error you get if you do an update of an exception list and someone else has changed it: ```ts { "message": "[exception-list:a08795b0-c85f-11ea-b1a6-c155df988a92]: version conflict, required seqNo [1074], primary term [1]. current document has seqNo [1077] and primary term [1]: [version_conflict_engine_exception] [exception-list:a08795b0-c85f-11ea-b1a6-c155df988a92]: version conflict, required seqNo [1074], primary term [1]. current document has seqNo [1077] and primary term [1], with { index_uuid=\"a2mgXBO6Tl2ULDq-MTs1Tw\" & shard=\"0\" & index=\".kibana-hassanabad_1\" }", "status_code": 409 } ``` Lists are the same way and flavor, they encode the _version the same way that saved objects do. To see those work you run these scripts: ```ts ./post_list.sh ./post_list_item.sh ./find_list.sh ./find_list_item.sh ``` ### Checklist - [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
FrankHassanabad
added a commit
to FrankHassanabad/kibana
that referenced
this pull request
Jul 20, 2020
…provements to lists (elastic#72337) ## Summary * Adds conflict versioning by exposing the "_version" from the saved object system. It renames "version" to "_version" so that we can use regular "version" later for versioning things for pre-packaged lists abilities. * Utilizes `t.OutputOf` in the requests and the data types to give us more correctly types * Removes the `Identity` utility as that is adding confusion and can confuse vs code rather than improves things * Removes extra types that were causing confusion which was an idiom from io-ts * Changes the wording of `Partial` by removing that and instead focuses the request types on either client side or server side at this point. NOTE: The UI can migrate to holding onto the `_version` and then push it back down when it wants to migrate to using the conflict resolution. If the UI does not push it down, then a value of undefined will be used which is indicating that no conflict errors are wanted. Output example of posting an exception list: ❯ ./post_exception_list.sh ```ts { "_tags": [ "endpoint", "process", "malware", "os:linux" ], "_version": "Wzk4NiwxXQ==", "created_at": "2020-07-17T18:59:22.872Z", "created_by": "yo", "description": "This is a sample endpoint type exception", "id": "a08795b0-c85f-11ea-b1a6-c155df988a92", "list_id": "simple_list", "name": "Sample Endpoint Exception List", "namespace_type": "single", "tags": [ "user added string for a tag", "malware" ], "tie_breaker_id": "b789ec05-3e0f-4344-a156-0c0f5b6e2f9c", "type": "detection", "updated_at": "2020-07-17T18:59:22.891Z", "updated_by": "yo" } ``` Output example of posting an exception list item ❯ ./post_exception_list_item.sh ```ts { "_tags": [ "endpoint", "process", "malware", "os:linux" ], "_version": "Wzk4NywxXQ==", "comments": [], "created_at": "2020-07-17T18:59:30.286Z", "created_by": "yo", "description": "This is a sample endpoint type exception", "entries": [ { "field": "actingProcess.file.signer", "operator": "excluded", "type": "exists" }, { "field": "host.name", "operator": "included", "type": "match_any", "value": [ "some host", "another host" ] } ], "id": "a4f2b800-c85f-11ea-b1a6-c155df988a92", "item_id": "simple_list_item", "list_id": "simple_list", "name": "Sample Endpoint Exception List", "namespace_type": "single", "tags": [ "user added string for a tag", "malware" ], "tie_breaker_id": "1dc456bc-7aa9-44b4-bca3-131689cf729f", "type": "simple", "updated_at": "2020-07-17T18:59:30.304Z", "updated_by": "yo" } ``` Output example of when you get an exception list: ❯ ./get_exception_list.sh simple_list ```ts { "_tags": [ "endpoint", "process", "malware", "os:linux" ], "_version": "WzEwNzcsMV0=", "created_at": "2020-07-17T18:59:22.872Z", "created_by": "yo", "description": "Different description", "id": "a08795b0-c85f-11ea-b1a6-c155df988a92", "list_id": "simple_list", "name": "Sample Endpoint Exception List", "namespace_type": "single", "tags": [ "user added string for a tag", "malware" ], "tie_breaker_id": "b789ec05-3e0f-4344-a156-0c0f5b6e2f9c", "type": "endpoint", "updated_at": "2020-07-17T20:01:24.958Z", "updated_by": "yo" } ``` Example of the error you get if you do an update of an exception list and someone else has changed it: ```ts { "message": "[exception-list:a08795b0-c85f-11ea-b1a6-c155df988a92]: version conflict, required seqNo [1074], primary term [1]. current document has seqNo [1077] and primary term [1]: [version_conflict_engine_exception] [exception-list:a08795b0-c85f-11ea-b1a6-c155df988a92]: version conflict, required seqNo [1074], primary term [1]. current document has seqNo [1077] and primary term [1], with { index_uuid=\"a2mgXBO6Tl2ULDq-MTs1Tw\" & shard=\"0\" & index=\".kibana-hassanabad_1\" }", "status_code": 409 } ``` Lists are the same way and flavor, they encode the _version the same way that saved objects do. To see those work you run these scripts: ```ts ./post_list.sh ./post_list_item.sh ./find_list.sh ./find_list_item.sh ``` ### Checklist - [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
gmmorris
added a commit
to gmmorris/kibana
that referenced
this pull request
Jul 20, 2020
* master: (60 commits) [SIEM][Detection Engine][Lists] Adds list permissions (elastic#72335) [SIEM][Detection Engine][Lists] Adds conflict versioning and io-ts improvements to lists (elastic#72337) [Resolver] no longer pass related event stats to process node component (elastic#72435) Revert "skip flaky suite (elastic#72146)" [Security Solution] Cleanup endpoint telemetry (elastic#71950) Unskip dashboard embeddable rendering tests (elastic#71824) [ENDPOINT] Added unerolling status for host. (elastic#72303) [Alerting][Connectors] Increase the size of the logos (elastic#72419) [SECURITY] [Timeline] Raw events not displayed (elastic#72387) [ML] Fixes display of regression stop stats if one is NaN (elastic#72412) [Ingest Pipelines] Processor Editor Move Tooltip (elastic#72239) Fix match phrase and not match phrase comparators (elastic#71850) [Plugin Generator] Generate tsconfig and useDefaultBehaviors (elastic#72040) [Security Solution][Timeline] Fix timeline styling and createFrom beh… (elastic#72152) [Resolver] Selector performance (elastic#72380) [Ingest Manager] Set `_meta` in the index.mappings (elastic#72026) [Ingest Manager] Do not bumb config revision during config creation (elastic#72270) [ML] Adding missing index pattern name to new job wizards (elastic#72400) [ML] improve annotation flyout performance (elastic#72299) [APM] Testing error rate API and restructuring folders (elastic#72257) ...
gmmorris
added a commit
to gmmorris/kibana
that referenced
this pull request
Jul 20, 2020
* master: (26 commits) [SIEM][Detection Engine][Lists] Adds list permissions (elastic#72335) [SIEM][Detection Engine][Lists] Adds conflict versioning and io-ts improvements to lists (elastic#72337) [Resolver] no longer pass related event stats to process node component (elastic#72435) Revert "skip flaky suite (elastic#72146)" [Security Solution] Cleanup endpoint telemetry (elastic#71950) Unskip dashboard embeddable rendering tests (elastic#71824) [ENDPOINT] Added unerolling status for host. (elastic#72303) [Alerting][Connectors] Increase the size of the logos (elastic#72419) [SECURITY] [Timeline] Raw events not displayed (elastic#72387) [ML] Fixes display of regression stop stats if one is NaN (elastic#72412) [Ingest Pipelines] Processor Editor Move Tooltip (elastic#72239) Fix match phrase and not match phrase comparators (elastic#71850) [Plugin Generator] Generate tsconfig and useDefaultBehaviors (elastic#72040) [Security Solution][Timeline] Fix timeline styling and createFrom beh… (elastic#72152) [Resolver] Selector performance (elastic#72380) [Ingest Manager] Set `_meta` in the index.mappings (elastic#72026) [Ingest Manager] Do not bumb config revision during config creation (elastic#72270) [ML] Adding missing index pattern name to new job wizards (elastic#72400) [ML] improve annotation flyout performance (elastic#72299) [APM] Testing error rate API and restructuring folders (elastic#72257) ...
FrankHassanabad
added a commit
that referenced
this pull request
Jul 20, 2020
…provements to lists (#72337) (#72484) ## Summary * Adds conflict versioning by exposing the "_version" from the saved object system. It renames "version" to "_version" so that we can use regular "version" later for versioning things for pre-packaged lists abilities. * Utilizes `t.OutputOf` in the requests and the data types to give us more correctly types * Removes the `Identity` utility as that is adding confusion and can confuse vs code rather than improves things * Removes extra types that were causing confusion which was an idiom from io-ts * Changes the wording of `Partial` by removing that and instead focuses the request types on either client side or server side at this point. NOTE: The UI can migrate to holding onto the `_version` and then push it back down when it wants to migrate to using the conflict resolution. If the UI does not push it down, then a value of undefined will be used which is indicating that no conflict errors are wanted. Output example of posting an exception list: ❯ ./post_exception_list.sh ```ts { "_tags": [ "endpoint", "process", "malware", "os:linux" ], "_version": "Wzk4NiwxXQ==", "created_at": "2020-07-17T18:59:22.872Z", "created_by": "yo", "description": "This is a sample endpoint type exception", "id": "a08795b0-c85f-11ea-b1a6-c155df988a92", "list_id": "simple_list", "name": "Sample Endpoint Exception List", "namespace_type": "single", "tags": [ "user added string for a tag", "malware" ], "tie_breaker_id": "b789ec05-3e0f-4344-a156-0c0f5b6e2f9c", "type": "detection", "updated_at": "2020-07-17T18:59:22.891Z", "updated_by": "yo" } ``` Output example of posting an exception list item ❯ ./post_exception_list_item.sh ```ts { "_tags": [ "endpoint", "process", "malware", "os:linux" ], "_version": "Wzk4NywxXQ==", "comments": [], "created_at": "2020-07-17T18:59:30.286Z", "created_by": "yo", "description": "This is a sample endpoint type exception", "entries": [ { "field": "actingProcess.file.signer", "operator": "excluded", "type": "exists" }, { "field": "host.name", "operator": "included", "type": "match_any", "value": [ "some host", "another host" ] } ], "id": "a4f2b800-c85f-11ea-b1a6-c155df988a92", "item_id": "simple_list_item", "list_id": "simple_list", "name": "Sample Endpoint Exception List", "namespace_type": "single", "tags": [ "user added string for a tag", "malware" ], "tie_breaker_id": "1dc456bc-7aa9-44b4-bca3-131689cf729f", "type": "simple", "updated_at": "2020-07-17T18:59:30.304Z", "updated_by": "yo" } ``` Output example of when you get an exception list: ❯ ./get_exception_list.sh simple_list ```ts { "_tags": [ "endpoint", "process", "malware", "os:linux" ], "_version": "WzEwNzcsMV0=", "created_at": "2020-07-17T18:59:22.872Z", "created_by": "yo", "description": "Different description", "id": "a08795b0-c85f-11ea-b1a6-c155df988a92", "list_id": "simple_list", "name": "Sample Endpoint Exception List", "namespace_type": "single", "tags": [ "user added string for a tag", "malware" ], "tie_breaker_id": "b789ec05-3e0f-4344-a156-0c0f5b6e2f9c", "type": "endpoint", "updated_at": "2020-07-17T20:01:24.958Z", "updated_by": "yo" } ``` Example of the error you get if you do an update of an exception list and someone else has changed it: ```ts { "message": "[exception-list:a08795b0-c85f-11ea-b1a6-c155df988a92]: version conflict, required seqNo [1074], primary term [1]. current document has seqNo [1077] and primary term [1]: [version_conflict_engine_exception] [exception-list:a08795b0-c85f-11ea-b1a6-c155df988a92]: version conflict, required seqNo [1074], primary term [1]. current document has seqNo [1077] and primary term [1], with { index_uuid=\"a2mgXBO6Tl2ULDq-MTs1Tw\" & shard=\"0\" & index=\".kibana-hassanabad_1\" }", "status_code": 409 } ``` Lists are the same way and flavor, they encode the _version the same way that saved objects do. To see those work you run these scripts: ```ts ./post_list.sh ./post_list_item.sh ./find_list.sh ./find_list_item.sh ``` ### Checklist - [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
FrankHassanabad
added a commit
that referenced
this pull request
Jul 20, 2020
…provements to lists (#72337) (#72483) ## Summary * Adds conflict versioning by exposing the "_version" from the saved object system. It renames "version" to "_version" so that we can use regular "version" later for versioning things for pre-packaged lists abilities. * Utilizes `t.OutputOf` in the requests and the data types to give us more correctly types * Removes the `Identity` utility as that is adding confusion and can confuse vs code rather than improves things * Removes extra types that were causing confusion which was an idiom from io-ts * Changes the wording of `Partial` by removing that and instead focuses the request types on either client side or server side at this point. NOTE: The UI can migrate to holding onto the `_version` and then push it back down when it wants to migrate to using the conflict resolution. If the UI does not push it down, then a value of undefined will be used which is indicating that no conflict errors are wanted. Output example of posting an exception list: ❯ ./post_exception_list.sh ```ts { "_tags": [ "endpoint", "process", "malware", "os:linux" ], "_version": "Wzk4NiwxXQ==", "created_at": "2020-07-17T18:59:22.872Z", "created_by": "yo", "description": "This is a sample endpoint type exception", "id": "a08795b0-c85f-11ea-b1a6-c155df988a92", "list_id": "simple_list", "name": "Sample Endpoint Exception List", "namespace_type": "single", "tags": [ "user added string for a tag", "malware" ], "tie_breaker_id": "b789ec05-3e0f-4344-a156-0c0f5b6e2f9c", "type": "detection", "updated_at": "2020-07-17T18:59:22.891Z", "updated_by": "yo" } ``` Output example of posting an exception list item ❯ ./post_exception_list_item.sh ```ts { "_tags": [ "endpoint", "process", "malware", "os:linux" ], "_version": "Wzk4NywxXQ==", "comments": [], "created_at": "2020-07-17T18:59:30.286Z", "created_by": "yo", "description": "This is a sample endpoint type exception", "entries": [ { "field": "actingProcess.file.signer", "operator": "excluded", "type": "exists" }, { "field": "host.name", "operator": "included", "type": "match_any", "value": [ "some host", "another host" ] } ], "id": "a4f2b800-c85f-11ea-b1a6-c155df988a92", "item_id": "simple_list_item", "list_id": "simple_list", "name": "Sample Endpoint Exception List", "namespace_type": "single", "tags": [ "user added string for a tag", "malware" ], "tie_breaker_id": "1dc456bc-7aa9-44b4-bca3-131689cf729f", "type": "simple", "updated_at": "2020-07-17T18:59:30.304Z", "updated_by": "yo" } ``` Output example of when you get an exception list: ❯ ./get_exception_list.sh simple_list ```ts { "_tags": [ "endpoint", "process", "malware", "os:linux" ], "_version": "WzEwNzcsMV0=", "created_at": "2020-07-17T18:59:22.872Z", "created_by": "yo", "description": "Different description", "id": "a08795b0-c85f-11ea-b1a6-c155df988a92", "list_id": "simple_list", "name": "Sample Endpoint Exception List", "namespace_type": "single", "tags": [ "user added string for a tag", "malware" ], "tie_breaker_id": "b789ec05-3e0f-4344-a156-0c0f5b6e2f9c", "type": "endpoint", "updated_at": "2020-07-17T20:01:24.958Z", "updated_by": "yo" } ``` Example of the error you get if you do an update of an exception list and someone else has changed it: ```ts { "message": "[exception-list:a08795b0-c85f-11ea-b1a6-c155df988a92]: version conflict, required seqNo [1074], primary term [1]. current document has seqNo [1077] and primary term [1]: [version_conflict_engine_exception] [exception-list:a08795b0-c85f-11ea-b1a6-c155df988a92]: version conflict, required seqNo [1074], primary term [1]. current document has seqNo [1077] and primary term [1], with { index_uuid=\"a2mgXBO6Tl2ULDq-MTs1Tw\" & shard=\"0\" & index=\".kibana-hassanabad_1\" }", "status_code": 409 } ``` Lists are the same way and flavor, they encode the _version the same way that saved objects do. To see those work you run these scripts: ```ts ./post_list.sh ./post_list_item.sh ./find_list.sh ./find_list_item.sh ``` ### Checklist - [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
gmmorris
added a commit
to gmmorris/kibana
that referenced
this pull request
Jul 21, 2020
…feature-privileges * alerting/consumer-based-rbac: (45 commits) fixed alerts test [SIEM][Detection Engine][Lists] Adds list permissions (elastic#72335) [SIEM][Detection Engine][Lists] Adds conflict versioning and io-ts improvements to lists (elastic#72337) [Resolver] no longer pass related event stats to process node component (elastic#72435) Revert "skip flaky suite (elastic#72146)" [Security Solution] Cleanup endpoint telemetry (elastic#71950) Unskip dashboard embeddable rendering tests (elastic#71824) [ENDPOINT] Added unerolling status for host. (elastic#72303) [Alerting][Connectors] Increase the size of the logos (elastic#72419) [SECURITY] [Timeline] Raw events not displayed (elastic#72387) [ML] Fixes display of regression stop stats if one is NaN (elastic#72412) [Ingest Pipelines] Processor Editor Move Tooltip (elastic#72239) Fix match phrase and not match phrase comparators (elastic#71850) [Plugin Generator] Generate tsconfig and useDefaultBehaviors (elastic#72040) [Security Solution][Timeline] Fix timeline styling and createFrom beh… (elastic#72152) allow user to disable alert even if they dont have privileges to the underlying action [Resolver] Selector performance (elastic#72380) [Ingest Manager] Set `_meta` in the index.mappings (elastic#72026) [Ingest Manager] Do not bumb config revision during config creation (elastic#72270) [ML] Adding missing index pattern name to new job wizards (elastic#72400) ...
1 task
Contributor
|
Pinging @elastic/security-solution (Team: SecuritySolution) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
t.OutputOfin the requests and the data types to give us more correctly typesIdentityutility as that is adding confusion and can confuse vs code rather than improves thingsPartialby removing that and instead focuses the request types on either client side or server side at this point.NOTE: The UI can migrate to holding onto the
_versionand then push it back down when it wants to migrate to using the conflict resolution. If the UI does not push it down, then a value of undefined will be used which is indicating that no conflict errors are wanted.Output example of posting an exception list:
❯ ./post_exception_list.sh
Output example of posting an exception list item
❯ ./post_exception_list_item.sh
Output example of when you get an exception list:
❯ ./get_exception_list.sh simple_list
Example of the error you get if you do an update of an exception list and someone else has changed it:
Lists are the same way and flavor, they encode the _version the same way that saved objects do. To see those work you run these scripts:
Checklist