[Security Solution] Test plan for upgrading prebuilt rules with preview (Rule Upgrade flyout) v1#205645
Conversation
775cb94 to
a7cb2ca
Compare
|
Pinging @elastic/security-detections-response (Team:Detections and Resp) |
|
Pinging @elastic/security-solution (Team: SecuritySolution) |
|
Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management) |
💚 Build Succeeded
cc @maximpn |
9f2c16c to
3616032
Compare
There was a problem hiding this comment.
Should we add a validation to check if the customization is correctly reverted after upgrade?
There was a problem hiding this comment.
@pborgonovi ++ we should cover this, I will address this in a follow-up PR.
|
@elasticmachine merge upstream |
17db057 to
56f0ab2
Compare
..._solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade_after_preview.md
Outdated
Show resolved
Hide resolved
..._solution/docs/testing/test_plans/detection_response/prebuilt_rules/upgrade_after_preview.md
Outdated
Show resolved
Hide resolved
|
|
||
| #### Preview non-customized field that has an upgrade (AAB) | ||
|
|
||
| **Automation**: Jest functional test for each \<field\>. |
There was a problem hiding this comment.
What's the difference between "Jest functional tests" and "Jest integration tests" mentioned below?
There was a problem hiding this comment.
These are the same. Ofc we should stick to the same wording. May we should use FrontEnd integration tests? Thanks for catching.
| Given an installed prebuilt rule | ||
| And that rule has no customizations | ||
| And <field> has an upgrade | ||
| When user opens the Rule Update Flyout | ||
| Then user should see <field> has no conflicts | ||
| And <field> is shown collapsed | ||
| When user expands <field> | ||
| Then user should see <field> Diff View | ||
| And user should see <field> Readonly View | ||
|
|
||
| Examples: | ||
| <field> = all customizable fields |
There was a problem hiding this comment.
Then user should see <field> has no conflicts
And <field> is shown collapsed
When user expands <field>
Then user should see <field> Diff View
And user should see <field> Readonly ViewI think we could elaborate more on the state of the flyout here:
- The field should be "accepted" and ready for upgrade
- The field value from the target rule version is selected as the final update and shown in the readonly view.
- Which options are available for selection in the diff view?
- It would be great to either explain what the mentioned views are in the Terminology section, or reword the text.
- What's the update status of the whole flyout?
- Is the Update CTA enabled?
I will address this in a follow-up PR.
| Given an installed prebuilt rule | ||
| And <field> is customized | ||
| And <field> has no upgrades | ||
| When user opens the Rule Update Flyout | ||
| Then user should see <field> has a customized value | ||
| And <field> has no conflicts | ||
| And <field> is shown collapsed | ||
| When user expands <field> | ||
| Then user should see <field> Diff View | ||
| And user should see <field> Readonly View | ||
|
|
||
| Examples: | ||
| <field> = all customizable fields |
There was a problem hiding this comment.
Similarly, in this and other tests from this section we could elaborate more on the state of the flyout.
| Given an installed prebuilt rule | ||
| And <field> is customized | ||
| And <field> has an upgrade resulting in a solvable conflict | ||
| When user opens the Rule Update Flyout | ||
| Then user should see <field> has a customized value | ||
| And <field> has a conflict | ||
| And <field> is shown expanded | ||
| And <field> Diff View is shown | ||
| And <field> Readonly View is shown | ||
| And <field> Readonly View displays a merged value | ||
| When user switches to edit form | ||
| Then user should see <field> edit form | ||
| And <field> edit form has merged value | ||
|
|
||
| Examples: <field> whose diff algo supports values merging | ||
| | data_source | | ||
| | tags | | ||
| | description | | ||
| | references | | ||
| | note | | ||
| | setup | | ||
| | threat_index | | ||
| | new_terms_fields | |
There was a problem hiding this comment.
In this test we should give concrete examples that would explain what solvable conflicts can look like, for a couple typical rule fields.
I'll address this in a follow-up PR.
|
Starting backport for target branches: 8.18, 8.x, 9.0 |
…ew (Rule Upgrade flyout) v1 (elastic#205645) **Epics:** elastic/security-team#1974 (internal), elastic#174168 **Addresses:** elastic#202078 ## Summary This PR adds a test plan for rule upgrade flyout. --------- Co-authored-by: Georgii Gorbachev <georgii.gorbachev@elastic.co> (cherry picked from commit ecd3316)
…ew (Rule Upgrade flyout) v1 (elastic#205645) **Epics:** elastic/security-team#1974 (internal), elastic#174168 **Addresses:** elastic#202078 ## Summary This PR adds a test plan for rule upgrade flyout. --------- Co-authored-by: Georgii Gorbachev <georgii.gorbachev@elastic.co> (cherry picked from commit ecd3316)
…ew (Rule Upgrade flyout) v1 (elastic#205645) **Epics:** elastic/security-team#1974 (internal), elastic#174168 **Addresses:** elastic#202078 ## Summary This PR adds a test plan for rule upgrade flyout. --------- Co-authored-by: Georgii Gorbachev <georgii.gorbachev@elastic.co> (cherry picked from commit ecd3316)
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
…h preview (Rule Upgrade flyout) v1 (#205645) (#211249) # Backport This will backport the following commits from `main` to `8.18`: - [[Security Solution] Test plan for upgrading prebuilt rules with preview (Rule Upgrade flyout) v1 (#205645)](#205645) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Maxim Palenov","email":"maxim.palenov@elastic.co"},"sourceCommit":{"committedDate":"2025-02-14T15:21:09Z","message":"[Security Solution] Test plan for upgrading prebuilt rules with preview (Rule Upgrade flyout) v1 (#205645)\n\n**Epics:** https://github.com/elastic/security-team/issues/1974\r\n(internal), https://github.com/elastic/kibana/issues/174168\r\n**Addresses:** https://github.com/elastic/kibana/issues/202078\r\n\r\n## Summary\r\n\r\nThis PR adds a test plan for rule upgrade flyout.\r\n\r\n---------\r\n\r\nCo-authored-by: Georgii Gorbachev <georgii.gorbachev@elastic.co>","sha":"ecd33168b152d0c268d4cec7881deedeb7628814","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","test-plan","v9.0.0","Team:Detections and Resp","Team: SecuritySolution","Team:Detection Rule Management","Feature:Prebuilt Detection Rules","backport:version","v8.18.0","v9.1.0","v8.19.0"],"title":"[Security Solution] Test plan for upgrading prebuilt rules with preview (Rule Upgrade flyout) v1","number":205645,"url":"https://github.com/elastic/kibana/pull/205645","mergeCommit":{"message":"[Security Solution] Test plan for upgrading prebuilt rules with preview (Rule Upgrade flyout) v1 (#205645)\n\n**Epics:** https://github.com/elastic/security-team/issues/1974\r\n(internal), https://github.com/elastic/kibana/issues/174168\r\n**Addresses:** https://github.com/elastic/kibana/issues/202078\r\n\r\n## Summary\r\n\r\nThis PR adds a test plan for rule upgrade flyout.\r\n\r\n---------\r\n\r\nCo-authored-by: Georgii Gorbachev <georgii.gorbachev@elastic.co>","sha":"ecd33168b152d0c268d4cec7881deedeb7628814"}},"sourceBranch":"main","suggestedTargetBranches":["9.0","8.18","8.x"],"targetPullRequestStates":[{"branch":"9.0","label":"v9.0.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.18","label":"v8.18.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/205645","number":205645,"mergeCommit":{"message":"[Security Solution] Test plan for upgrading prebuilt rules with preview (Rule Upgrade flyout) v1 (#205645)\n\n**Epics:** https://github.com/elastic/security-team/issues/1974\r\n(internal), https://github.com/elastic/kibana/issues/174168\r\n**Addresses:** https://github.com/elastic/kibana/issues/202078\r\n\r\n## Summary\r\n\r\nThis PR adds a test plan for rule upgrade flyout.\r\n\r\n---------\r\n\r\nCo-authored-by: Georgii Gorbachev <georgii.gorbachev@elastic.co>","sha":"ecd33168b152d0c268d4cec7881deedeb7628814"}},{"branch":"8.x","label":"v8.19.0","branchLabelMappingKey":"^v8.19.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Maxim Palenov <maxim.palenov@elastic.co>
… preview (Rule Upgrade flyout) v1 (#205645) (#211251) # Backport This will backport the following commits from `main` to `9.0`: - [[Security Solution] Test plan for upgrading prebuilt rules with preview (Rule Upgrade flyout) v1 (#205645)](#205645) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Maxim Palenov","email":"maxim.palenov@elastic.co"},"sourceCommit":{"committedDate":"2025-02-14T15:21:09Z","message":"[Security Solution] Test plan for upgrading prebuilt rules with preview (Rule Upgrade flyout) v1 (#205645)\n\n**Epics:** https://github.com/elastic/security-team/issues/1974\r\n(internal), https://github.com/elastic/kibana/issues/174168\r\n**Addresses:** https://github.com/elastic/kibana/issues/202078\r\n\r\n## Summary\r\n\r\nThis PR adds a test plan for rule upgrade flyout.\r\n\r\n---------\r\n\r\nCo-authored-by: Georgii Gorbachev <georgii.gorbachev@elastic.co>","sha":"ecd33168b152d0c268d4cec7881deedeb7628814","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","test-plan","v9.0.0","Team:Detections and Resp","Team: SecuritySolution","Team:Detection Rule Management","Feature:Prebuilt Detection Rules","backport:version","v8.18.0","v9.1.0","v8.19.0"],"title":"[Security Solution] Test plan for upgrading prebuilt rules with preview (Rule Upgrade flyout) v1","number":205645,"url":"https://github.com/elastic/kibana/pull/205645","mergeCommit":{"message":"[Security Solution] Test plan for upgrading prebuilt rules with preview (Rule Upgrade flyout) v1 (#205645)\n\n**Epics:** https://github.com/elastic/security-team/issues/1974\r\n(internal), https://github.com/elastic/kibana/issues/174168\r\n**Addresses:** https://github.com/elastic/kibana/issues/202078\r\n\r\n## Summary\r\n\r\nThis PR adds a test plan for rule upgrade flyout.\r\n\r\n---------\r\n\r\nCo-authored-by: Georgii Gorbachev <georgii.gorbachev@elastic.co>","sha":"ecd33168b152d0c268d4cec7881deedeb7628814"}},"sourceBranch":"main","suggestedTargetBranches":["9.0","8.18","8.x"],"targetPullRequestStates":[{"branch":"9.0","label":"v9.0.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.18","label":"v8.18.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/205645","number":205645,"mergeCommit":{"message":"[Security Solution] Test plan for upgrading prebuilt rules with preview (Rule Upgrade flyout) v1 (#205645)\n\n**Epics:** https://github.com/elastic/security-team/issues/1974\r\n(internal), https://github.com/elastic/kibana/issues/174168\r\n**Addresses:** https://github.com/elastic/kibana/issues/202078\r\n\r\n## Summary\r\n\r\nThis PR adds a test plan for rule upgrade flyout.\r\n\r\n---------\r\n\r\nCo-authored-by: Georgii Gorbachev <georgii.gorbachev@elastic.co>","sha":"ecd33168b152d0c268d4cec7881deedeb7628814"}},{"branch":"8.x","label":"v8.19.0","branchLabelMappingKey":"^v8.19.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Maxim Palenov <maxim.palenov@elastic.co>
… preview (Rule Upgrade flyout) v1 (#205645) (#211250) # Backport This will backport the following commits from `main` to `8.x`: - [[Security Solution] Test plan for upgrading prebuilt rules with preview (Rule Upgrade flyout) v1 (#205645)](#205645) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Maxim Palenov","email":"maxim.palenov@elastic.co"},"sourceCommit":{"committedDate":"2025-02-14T15:21:09Z","message":"[Security Solution] Test plan for upgrading prebuilt rules with preview (Rule Upgrade flyout) v1 (#205645)\n\n**Epics:** https://github.com/elastic/security-team/issues/1974\r\n(internal), https://github.com/elastic/kibana/issues/174168\r\n**Addresses:** https://github.com/elastic/kibana/issues/202078\r\n\r\n## Summary\r\n\r\nThis PR adds a test plan for rule upgrade flyout.\r\n\r\n---------\r\n\r\nCo-authored-by: Georgii Gorbachev <georgii.gorbachev@elastic.co>","sha":"ecd33168b152d0c268d4cec7881deedeb7628814","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","test-plan","v9.0.0","Team:Detections and Resp","Team: SecuritySolution","Team:Detection Rule Management","Feature:Prebuilt Detection Rules","backport:version","v8.18.0","v9.1.0","v8.19.0"],"title":"[Security Solution] Test plan for upgrading prebuilt rules with preview (Rule Upgrade flyout) v1","number":205645,"url":"https://github.com/elastic/kibana/pull/205645","mergeCommit":{"message":"[Security Solution] Test plan for upgrading prebuilt rules with preview (Rule Upgrade flyout) v1 (#205645)\n\n**Epics:** https://github.com/elastic/security-team/issues/1974\r\n(internal), https://github.com/elastic/kibana/issues/174168\r\n**Addresses:** https://github.com/elastic/kibana/issues/202078\r\n\r\n## Summary\r\n\r\nThis PR adds a test plan for rule upgrade flyout.\r\n\r\n---------\r\n\r\nCo-authored-by: Georgii Gorbachev <georgii.gorbachev@elastic.co>","sha":"ecd33168b152d0c268d4cec7881deedeb7628814"}},"sourceBranch":"main","suggestedTargetBranches":["9.0","8.18","8.x"],"targetPullRequestStates":[{"branch":"9.0","label":"v9.0.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.18","label":"v8.18.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/205645","number":205645,"mergeCommit":{"message":"[Security Solution] Test plan for upgrading prebuilt rules with preview (Rule Upgrade flyout) v1 (#205645)\n\n**Epics:** https://github.com/elastic/security-team/issues/1974\r\n(internal), https://github.com/elastic/kibana/issues/174168\r\n**Addresses:** https://github.com/elastic/kibana/issues/202078\r\n\r\n## Summary\r\n\r\nThis PR adds a test plan for rule upgrade flyout.\r\n\r\n---------\r\n\r\nCo-authored-by: Georgii Gorbachev <georgii.gorbachev@elastic.co>","sha":"ecd33168b152d0c268d4cec7881deedeb7628814"}},{"branch":"8.x","label":"v8.19.0","branchLabelMappingKey":"^v8.19.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Maxim Palenov <maxim.palenov@elastic.co>
…ew (Rule Upgrade flyout) v1 (elastic#205645) **Epics:** elastic/security-team#1974 (internal), elastic#174168 **Addresses:** elastic#202078 ## Summary This PR adds a test plan for rule upgrade flyout. --------- Co-authored-by: Georgii Gorbachev <georgii.gorbachev@elastic.co>
…le Upgrade Flyout) (#210377) **Partially addresses:** #205645 ## Summary This PR implements Frontend integration tests from the [upgrading prebuilt rules one-by-one with preview test plan](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_upgrade_with_preview.md). ## Details This PR add Jest integration tests (`@kbn/test/jest_integration` preset) for Rule Upgrade Flyout. Test scenarios are described in [upgrading prebuilt rules one-by-one with preview test plan](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_upgrade_with_preview.md). Tests cover each `diffable rule` field separately to guarantee visibility on broken functionality. `esql_query` and `threat_mapping` fields were skipped due to mocking difficulties. ### Tests setup - Rules Management page is used as the root component to test functionality in integration - HTTP responses are mocked via mocking return values from `Kibana.http.fetch()` method - Test scenarios are the same for each diffable rule field and moved out to reusable utility functions
…le Upgrade Flyout) (elastic#210377) **Partially addresses:** elastic#205645 ## Summary This PR implements Frontend integration tests from the [upgrading prebuilt rules one-by-one with preview test plan](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_upgrade_with_preview.md). ## Details This PR add Jest integration tests (`@kbn/test/jest_integration` preset) for Rule Upgrade Flyout. Test scenarios are described in [upgrading prebuilt rules one-by-one with preview test plan](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_upgrade_with_preview.md). Tests cover each `diffable rule` field separately to guarantee visibility on broken functionality. `esql_query` and `threat_mapping` fields were skipped due to mocking difficulties. ### Tests setup - Rules Management page is used as the root component to test functionality in integration - HTTP responses are mocked via mocking return values from `Kibana.http.fetch()` method - Test scenarios are the same for each diffable rule field and moved out to reusable utility functions (cherry picked from commit ea7ccca)
…le Upgrade Flyout) (elastic#210377) **Partially addresses:** elastic#205645 ## Summary This PR implements Frontend integration tests from the [upgrading prebuilt rules one-by-one with preview test plan](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_upgrade_with_preview.md). ## Details This PR add Jest integration tests (`@kbn/test/jest_integration` preset) for Rule Upgrade Flyout. Test scenarios are described in [upgrading prebuilt rules one-by-one with preview test plan](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_upgrade_with_preview.md). Tests cover each `diffable rule` field separately to guarantee visibility on broken functionality. `esql_query` and `threat_mapping` fields were skipped due to mocking difficulties. ### Tests setup - Rules Management page is used as the root component to test functionality in integration - HTTP responses are mocked via mocking return values from `Kibana.http.fetch()` method - Test scenarios are the same for each diffable rule field and moved out to reusable utility functions
…le Upgrade Flyout) (elastic#210377) **Partially addresses:** elastic#205645 ## Summary This PR implements Frontend integration tests from the [upgrading prebuilt rules one-by-one with preview test plan](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_upgrade_with_preview.md). ## Details This PR add Jest integration tests (`@kbn/test/jest_integration` preset) for Rule Upgrade Flyout. Test scenarios are described in [upgrading prebuilt rules one-by-one with preview test plan](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_upgrade_with_preview.md). Tests cover each `diffable rule` field separately to guarantee visibility on broken functionality. `esql_query` and `threat_mapping` fields were skipped due to mocking difficulties. ### Tests setup - Rules Management page is used as the root component to test functionality in integration - HTTP responses are mocked via mocking return values from `Kibana.http.fetch()` method - Test scenarios are the same for each diffable rule field and moved out to reusable utility functions (cherry picked from commit ea7ccca) # Conflicts: # x-pack/solutions/security/plugins/security_solution/public/common/lib/kibana/kibana_react.mock.ts
Epics: https://github.com/elastic/security-team/issues/1974 (internal), #174168
Addresses: #202078
Summary
This PR adds a test plan for rule upgrade flyout.