Commit d28a6dd
committed
[Security Solution] Implement prebuilt rules upgrade test plan (#228620)
**Addresses: #202078
## Summary
This PR implements Prebuilt Rules Upgrade [with (via Prebuilt Rule Upgrade Flyout)](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_upgrade_with_preview.md) and [without (via Prebuilt Rules Upgrade table)](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_upgrade_without_preview.md) preview test plans.
## Details
Since some of the test scenarios had been implemented earlier this PR involves refactoring as well and tests reorganization. The following has been done in the scope
- Prebuilt Rules Installation and Prebuilt Rules Upgrade workflows in Cypress have been split
- Existing tests were renamed to better match to the test plans
- Prebuilt Rules Upgrade Cypress tests were reorganized to
- `upgrade_with_preview.cy.ts` - Upgrade via Prebuilt Rule Upgrade Flyout
- `upgrade_without_preview.cy.ts` - Upgrade via via Prebuilt Rules Upgrade table
- Missing tests have been added to the corresponding files for Jest Integration and Cypress tests
## Not implemented test scenarios
The following test scenarios weren't implemented due to time constraints
- [Prebuilt rules upgrade with preview test plan](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_upgrade_with_preview.md#concurrency-control) (Upgrade via Prebuilt Rule Upgrade Flyout)
- **Concurrency control** e2e tests
The tests verify that a toast message appears after either prebuilt rule has been edited (revision bump) or a new prebuilt rule version is available (version bump, a new prebuilt rules package version has been installed). Since UI refetches data every 5 minutes e2e tests either has to wait for ~5 minutes for toasts to appear or use `cy.clock()` to mock timers. The latter doesn't work in a simple way and requires time for deeper investigation.
But there are [unit tests](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/security_solution/public/detection_engine/rule_management_ui/components/rules_table/upgrade_prebuilt_rules_table/use_prebuilt_rules_upgrade_state.test.ts) for concurrency control.
- [Prebuilt rules upgrade without preview test plan](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/plugins/security_solution/docs/testing/test_plans/detection_response/prebuilt_rules/prebuilt_rule_upgrade_without_preview.md) (Prebuilt Rules Bulk upgrade)
- **Rule upgrade workflow: filtering, sorting, pagination**
There are two existing table filtering tests but besides that no new tests has been added. These tests should be covered in the scope of #166215.
- **Rule upgrade workflow: Edge cases**/**Scenario: Rule bound data is preserved after upgrading a rule to a newer version**
- **Error handling**/**Scenario: Error is handled when any upgrade operation on prebuilt rules fails**
- **Licensing: API endpoints**
## Flaky test runner
- ✅ [e2e Cypress tests Rule Management group](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8812) (100 runs)
(cherry picked from commit 67006ad)1 parent ac4942c commit d28a6dd
67 files changed
Lines changed: 4590 additions & 2051 deletions
File tree
- x-pack
- solutions/security/plugins/security_solution
- docs/testing/test_plans/detection_response/prebuilt_rules
- public
- common/test/eui
- detection_engine
- rule_creation_ui/components/step_define_rule
- rule_management_ui
- components/rules_table/upgrade_prebuilt_rules_table/use_upgrade_with_conflicts_modal
- pages/rule_management/__integration_tests__/rules_upgrade
- test_utils
- upgrade_rule_after_preview
- common_fields
- type_specific_fields
- rule_management/components/rule_details
- json_diff
- three_way_diff
- comparison_side
- field_final_side/components
- final_edit/fields/alert_suppression
- test/security_solution_cypress/cypress
- e2e/detection_response
- detection_engine/detection_alerts/assignments
- rule_management/prebuilt_rules
- screens
- tasks
- prebuilt_rules
- urls
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 63 additions & 22 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
30 | 30 | | |
31 | 31 | | |
32 | 32 | | |
33 | | - | |
34 | 33 | | |
35 | 34 | | |
36 | 35 | | |
| |||
51 | 50 | | |
52 | 51 | | |
53 | 52 | | |
54 | | - | |
| 53 | + | |
55 | 54 | | |
56 | 55 | | |
57 | 56 | | |
| |||
61 | 60 | | |
62 | 61 | | |
63 | 62 | | |
| 63 | + | |
| 64 | + | |
64 | 65 | | |
65 | 66 | | |
66 | 67 | | |
67 | 68 | | |
68 | | - | |
69 | | - | |
| 69 | + | |
| 70 | + | |
70 | 71 | | |
71 | 72 | | |
72 | 73 | | |
| |||
144 | 145 | | |
145 | 146 | | |
146 | 147 | | |
| 148 | + | |
| 149 | + | |
147 | 150 | | |
148 | 151 | | |
149 | 152 | | |
| |||
152 | 155 | | |
153 | 156 | | |
154 | 157 | | |
155 | | - | |
156 | | - | |
157 | | - | |
158 | | - | |
159 | | - | |
160 | | - | |
161 | | - | |
162 | | - | |
163 | | - | |
164 | | - | |
165 | | - | |
166 | | - | |
167 | | - | |
168 | | - | |
169 | 158 | | |
170 | 159 | | |
171 | 160 | | |
| |||
515 | 504 | | |
516 | 505 | | |
517 | 506 | | |
518 | | - | |
| 507 | + | |
519 | 508 | | |
520 | 509 | | |
521 | 510 | | |
522 | | - | |
| 511 | + | |
523 | 512 | | |
524 | 513 | | |
525 | 514 | | |
| |||
744 | 733 | | |
745 | 734 | | |
746 | 735 | | |
| 736 | + | |
| 737 | + | |
| 738 | + | |
| 739 | + | |
| 740 | + | |
| 741 | + | |
| 742 | + | |
| 743 | + | |
| 744 | + | |
| 745 | + | |
| 746 | + | |
| 747 | + | |
| 748 | + | |
| 749 | + | |
| 750 | + | |
| 751 | + | |
| 752 | + | |
| 753 | + | |
| 754 | + | |
| 755 | + | |
| 756 | + | |
| 757 | + | |
| 758 | + | |
| 759 | + | |
| 760 | + | |
| 761 | + | |
| 762 | + | |
| 763 | + | |
| 764 | + | |
| 765 | + | |
| 766 | + | |
| 767 | + | |
| 768 | + | |
| 769 | + | |
| 770 | + | |
| 771 | + | |
| 772 | + | |
| 773 | + | |
| 774 | + | |
| 775 | + | |
| 776 | + | |
| 777 | + | |
| 778 | + | |
| 779 | + | |
| 780 | + | |
| 781 | + | |
| 782 | + | |
| 783 | + | |
747 | 784 | | |
748 | 785 | | |
749 | 786 | | |
| |||
787 | 824 | | |
788 | 825 | | |
789 | 826 | | |
790 | | - | |
| 827 | + | |
| 828 | + | |
| 829 | + | |
| 830 | + | |
| 831 | + | |
791 | 832 | | |
792 | 833 | | |
793 | 834 | | |
| |||
800 | 841 | | |
801 | 842 | | |
802 | 843 | | |
803 | | - | |
| 844 | + | |
804 | 845 | | |
805 | 846 | | |
806 | 847 | | |
| |||
Lines changed: 2 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
119 | 119 | | |
120 | 120 | | |
121 | 121 | | |
| 122 | + | |
| 123 | + | |
122 | 124 | | |
123 | 125 | | |
124 | 126 | | |
| |||
Lines changed: 20 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
11 | 19 | | |
12 | 20 | | |
13 | 21 | | |
| |||
39 | 47 | | |
40 | 48 | | |
41 | 49 | | |
42 | | - | |
| 50 | + | |
43 | 51 | | |
44 | 52 | | |
45 | 53 | | |
| |||
60 | 68 | | |
61 | 69 | | |
62 | 70 | | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
63 | 82 | | |
64 | 83 | | |
65 | 84 | | |
| |||
Lines changed: 4 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
72 | 72 | | |
73 | 73 | | |
74 | 74 | | |
75 | | - | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
76 | 79 | | |
77 | 80 | | |
78 | 81 | | |
| |||
Lines changed: 18 additions & 14 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
265 | 265 | | |
266 | 266 | | |
267 | 267 | | |
| 268 | + | |
268 | 269 | | |
269 | 270 | | |
270 | 271 | | |
| |||
273 | 274 | | |
274 | 275 | | |
275 | 276 | | |
| 277 | + | |
276 | 278 | | |
277 | 279 | | |
278 | 280 | | |
| |||
308 | 310 | | |
309 | 311 | | |
310 | 312 | | |
311 | | - | |
312 | | - | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
313 | 316 | | |
314 | 317 | | |
315 | 318 | | |
316 | | - | |
317 | | - | |
318 | | - | |
319 | | - | |
320 | | - | |
321 | | - | |
322 | | - | |
323 | | - | |
324 | | - | |
325 | | - | |
326 | | - | |
327 | | - | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
328 | 332 | | |
329 | 333 | | |
330 | 334 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
32 | 32 | | |
33 | 33 | | |
34 | 34 | | |
35 | | - | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
36 | 41 | | |
37 | 42 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
25 | 25 | | |
26 | 26 | | |
27 | 27 | | |
28 | | - | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
29 | 33 | | |
30 | 34 | | |
31 | 35 | | |
| |||
41 | 45 | | |
42 | 46 | | |
43 | 47 | | |
44 | | - | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
45 | 53 | | |
46 | 54 | | |
47 | 55 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
41 | 41 | | |
42 | 42 | | |
43 | 43 | | |
44 | | - | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
45 | 50 | | |
46 | 51 | | |
47 | 52 | | |
| |||
57 | 62 | | |
58 | 63 | | |
59 | 64 | | |
| 65 | + | |
60 | 66 | | |
61 | 67 | | |
62 | 68 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
| 8 | + | |
8 | 9 | | |
9 | 10 | | |
10 | 11 | | |
11 | 12 | | |
12 | 13 | | |
| 14 | + | |
| 15 | + | |
13 | 16 | | |
| 17 | + | |
14 | 18 | | |
15 | | - | |
16 | 19 | | |
17 | 20 | | |
18 | 21 | | |
| |||
31 | 34 | | |
32 | 35 | | |
33 | 36 | | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
34 | 44 | | |
35 | 45 | | |
36 | 46 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
65 | 65 | | |
66 | 66 | | |
67 | 67 | | |
68 | | - | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
69 | 72 | | |
70 | 73 | | |
71 | 74 | | |
72 | 75 | | |
73 | | - | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
74 | 81 | | |
75 | 82 | | |
76 | 83 | | |
77 | 84 | | |
78 | 85 | | |
79 | | - | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
80 | 89 | | |
81 | 90 | | |
82 | 91 | | |
| |||
0 commit comments