Skip to content

[Security Solution] Fix flaky test: detection_rules/bulk_edit_rules_actions.cy.ts#163698

Merged
jpdjere merged 11 commits intoelastic:mainfrom
jpdjere:154721-fix-flaky-cy-test-bulk-actions
Aug 16, 2023
Merged

[Security Solution] Fix flaky test: detection_rules/bulk_edit_rules_actions.cy.ts#163698
jpdjere merged 11 commits intoelastic:mainfrom
jpdjere:154721-fix-flaky-cy-test-bulk-actions

Conversation

@jpdjere
Copy link
Copy Markdown
Contributor

@jpdjere jpdjere commented Aug 11, 2023

Fixes: #154721

Summary

  • Fixes flaky test: x-pack/plugins/security_solution/cypress/e2e/detection_response/rule_management/rule_actions/bulk_actions/bulk_edit_rules_actions.cy.ts
  • Test title: Detection rules, bulk edit of rule actions

Details

For: Detection rules, bulk edit of rule actions - Restricted action privileges - User with no privileges can't add rule actions

  • Since this test logs in with a user with missing privileges, the "Missing privileges" callout is shown at the top of the Rules Table. The selectNumberOfRules(); command selects rules one by one by clicking on their checkboxes. However, flakiness was caused when the callout was rendered while the selection of the rules was happening, causing a layout shift that caused the selection of a checkbox to lose focus, and not being able to be checked. This was solved by waiting the callout to be rendered before the selection of rules start, with the new waitForCallOutToBeShownmethod.

For: Detection rules, bulk edit of rule actions - All actions privileges - before/beforeEach Clause

  • Tests were failing in the beforeEach clause because the first test, mentioned above, would be logged in with a ROLES.hunter_no_actions role, and logging in with a user with permissions happened in a before clause instead of a beforeEach clause. This caused the rest of the suite to continue with a role without permissions, and the setup of the second test would fail as the API requests done would fail with 401. Moving the initial logging-in from a before clause to a beforeEach clause solved this issue.

For: Detection rules, bulk edit of rule actions - All actions privileges - Add a rule action to rules (existing connector)

  • This flakiness was extremely rare, but could be reproduced after about 400 iterations. It was caused by a similar reason as the first case above: while rules were being selected one by one, the table would auto refresh and focus would be lost from the checkbox that was about to be selected. This aws fixed by disabling autorefresh in the setup.

Other changes

  • Prevents the installation of security_detection_engine package and creates mock rules instead.
  • Creates the waitForCallOutToBeShown method and moves the callout IDs spread across different files to a a single file where they are exported from.

@jpdjere jpdjere self-assigned this Aug 11, 2023
@jpdjere
Copy link
Copy Markdown
Contributor Author

jpdjere commented Aug 11, 2023

@jpdjere
Copy link
Copy Markdown
Contributor Author

jpdjere commented Aug 11, 2023

@jpdjere
Copy link
Copy Markdown
Contributor Author

jpdjere commented Aug 11, 2023

@jpdjere jpdjere force-pushed the 154721-fix-flaky-cy-test-bulk-actions branch from e82b22f to 5084e3c Compare August 14, 2023 10:20
@jpdjere
Copy link
Copy Markdown
Contributor Author

jpdjere commented Aug 14, 2023

[3rd change] Fix flaky test: detection_rules/bulk_edit_rules_actions.cy.ts #163698
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2867

@jpdjere
Copy link
Copy Markdown
Contributor Author

jpdjere commented Aug 14, 2023

[Fix: Disable autorefresh] Fix flaky test: detection_rules/bulk_edit_rules_actions.cy.ts #163698
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2868

@jpdjere jpdjere added test release_note:skip Skip the PR/issue when compiling release notes backport:skip This PR does not require backporting Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Detection Rule Management Security Detection Rule Management Team v8.10.0 labels Aug 14, 2023
@jpdjere jpdjere requested a review from maximpn August 14, 2023 17:01
@jpdjere jpdjere marked this pull request as ready for review August 14, 2023 17:03
@jpdjere jpdjere requested review from a team as code owners August 14, 2023 17:03
@elasticmachine
Copy link
Copy Markdown
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@elasticmachine
Copy link
Copy Markdown
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

Copy link
Copy Markdown
Contributor

@e40pud e40pud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@banderror banderror removed the backport:skip This PR does not require backporting label Aug 15, 2023
@jpdjere jpdjere force-pushed the 154721-fix-flaky-cy-test-bulk-actions branch from fbe0a54 to ac09363 Compare August 16, 2023 08:04
@jpdjere jpdjere requested a review from a team as a code owner August 16, 2023 08:04
Copy link
Copy Markdown
Contributor

@maximpn maximpn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

@maximpn
Copy link
Copy Markdown
Contributor

maximpn commented Aug 16, 2023

@elasticmachine merge upstream

@kibana-ci
Copy link
Copy Markdown

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @jpdjere

@jpdjere jpdjere merged commit d725c28 into elastic:main Aug 16, 2023
@kibanamachine kibanamachine added the backport:skip This PR does not require backporting label Aug 16, 2023
@jpdjere jpdjere deleted the 154721-fix-flaky-cy-test-bulk-actions branch August 16, 2023 12:45
hop-dev pushed a commit to hop-dev/kibana that referenced this pull request Aug 16, 2023
…actions.cy.ts` (elastic#163698)

Fixes: elastic#154721

## Summary

- Fixes flaky test:
`x-pack/plugins/security_solution/cypress/e2e/detection_response/rule_management/rule_actions/bulk_actions/bulk_edit_rules_actions.cy.ts`
- Test title: `Detection rules, bulk edit of rule actions`


## Details

For: `Detection rules, bulk edit of rule actions - Restricted action
privileges - User with no privileges can't add rule actions`
- Since this test logs in with a user with missing privileges, the
"Missing privileges" callout is shown at the top of the Rules Table. The
`selectNumberOfRules();` command selects rules one by one by clicking on
their checkboxes. However, flakiness was caused when the callout was
rendered while the selection of the rules was happening, causing a
layout shift that caused the selection of a checkbox to lose focus, and
not being able to be checked. This was solved by waiting the callout to
be rendered before the selection of rules start, with the new
`waitForCallOutToBeShown`method.

For: `Detection rules, bulk edit of rule actions - All actions
privileges - before/beforeEach Clause`
- Tests were failing in the `beforeEach` clause because the first test,
mentioned above, would be logged in with a `ROLES.hunter_no_actions`
role, and logging in with a user with permissions happened in a `before`
clause instead of a `beforeEach` clause. This caused the rest of the
suite to continue with a role without permissions, and the setup of the
second test would fail as the API requests done would fail with `401`.
Moving the initial logging-in from a `before` clause to a `beforeEach`
clause solved this issue.

For: `Detection rules, bulk edit of rule actions - All actions
privileges - Add a rule action to rules (existing connector)`
- This flakiness was extremely rare, but could be reproduced after about
400 iterations. It was caused by a similar reason as the first case
above: while rules were being selected one by one, the table would auto
refresh and focus would be lost from the checkbox that was about to be
selected. This aws fixed by disabling autorefresh in the setup.

### Other changes
- Prevents the installation of `security_detection_engine` package and
creates mock rules instead.
- Creates the `waitForCallOutToBeShown` method and moves the callout IDs
spread across different files to a a single file where they are exported
from.

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
e40pud added a commit that referenced this pull request Aug 21, 2023
…ewer cypress test (#164283)

## Summary

Fixes: #162569
Fixes: #164061
Fixes: #164058
Fixes: #163546
Fixes: #162669

We tried to fix the issue with this PR
#162839 but test failed again.

This is another attempt to fix it using the @jpdjere's approach where we
disable rule's table refreshing
(#163698).
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Aug 21, 2023
…ewer cypress test (elastic#164283)

## Summary

Fixes: elastic#162569
Fixes: elastic#164061
Fixes: elastic#164058
Fixes: elastic#163546
Fixes: elastic#162669

We tried to fix the issue with this PR
elastic#162839 but test failed again.

This is another attempt to fix it using the @jpdjere's approach where we
disable rule's table refreshing
(elastic#163698).

(cherry picked from commit 4477f64)
kibanamachine referenced this pull request Aug 21, 2023
…only viewer cypress test (#164283) (#164346)

# Backport

This will backport the following commits from `main` to `8.10`:
- [[Security Solution] [Detections] Fixes flakey exceptions read-only
viewer cypress test
(#164283)](#164283)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Ievgen
Sorokopud","email":"ievgen.sorokopud@elastic.co"},"sourceCommit":{"committedDate":"2023-08-21T18:40:14Z","message":"[Security
Solution] [Detections] Fixes flakey exceptions read-only viewer cypress
test (#164283)\n\n## Summary\r\n\r\nFixes:
https://github.com/elastic/kibana/issues/162569\r\nFixes:
https://github.com/elastic/kibana/issues/164061\r\nFixes:
https://github.com/elastic/kibana/issues/164058\r\nFixes:
https://github.com/elastic/kibana/issues/163546\r\nFixes:
https://github.com/elastic/kibana/issues/162669\r\n\r\nWe tried to fix
the issue with this PR\r\nhttps://github.com//pull/162839
but test failed again.\r\n\r\nThis is another attempt to fix it using
the @jpdjere's approach where we\r\ndisable rule's table
refreshing\r\n(https://github.com/elastic/kibana/pull/163698).","sha":"4477f642e3297355ef676dcf485efb0cb49c4fcb","branchLabelMapping":{"^v8.11.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:
SecuritySolution","Team:Detection
Engine","v8.10.0","v8.11.0"],"number":164283,"url":"https://github.com/elastic/kibana/pull/164283","mergeCommit":{"message":"[Security
Solution] [Detections] Fixes flakey exceptions read-only viewer cypress
test (#164283)\n\n## Summary\r\n\r\nFixes:
https://github.com/elastic/kibana/issues/162569\r\nFixes:
https://github.com/elastic/kibana/issues/164061\r\nFixes:
https://github.com/elastic/kibana/issues/164058\r\nFixes:
https://github.com/elastic/kibana/issues/163546\r\nFixes:
https://github.com/elastic/kibana/issues/162669\r\n\r\nWe tried to fix
the issue with this PR\r\nhttps://github.com//pull/162839
but test failed again.\r\n\r\nThis is another attempt to fix it using
the @jpdjere's approach where we\r\ndisable rule's table
refreshing\r\n(https://github.com/elastic/kibana/pull/163698).","sha":"4477f642e3297355ef676dcf485efb0cb49c4fcb"}},"sourceBranch":"main","suggestedTargetBranches":["8.10"],"targetPullRequestStates":[{"branch":"8.10","label":"v8.10.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.11.0","labelRegex":"^v8.11.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/164283","number":164283,"mergeCommit":{"message":"[Security
Solution] [Detections] Fixes flakey exceptions read-only viewer cypress
test (#164283)\n\n## Summary\r\n\r\nFixes:
https://github.com/elastic/kibana/issues/162569\r\nFixes:
https://github.com/elastic/kibana/issues/164061\r\nFixes:
https://github.com/elastic/kibana/issues/164058\r\nFixes:
https://github.com/elastic/kibana/issues/163546\r\nFixes:
https://github.com/elastic/kibana/issues/162669\r\n\r\nWe tried to fix
the issue with this PR\r\nhttps://github.com//pull/162839
but test failed again.\r\n\r\nThis is another attempt to fix it using
the @jpdjere's approach where we\r\ndisable rule's table
refreshing\r\n(https://github.com/elastic/kibana/pull/163698).","sha":"4477f642e3297355ef676dcf485efb0cb49c4fcb"}}]}]
BACKPORT-->

Co-authored-by: Ievgen Sorokopud <ievgen.sorokopud@elastic.co>
e40pud added a commit to e40pud/kibana that referenced this pull request Aug 21, 2023
…ewer cypress test (elastic#164283)

## Summary

Fixes: elastic#162569
Fixes: elastic#164061
Fixes: elastic#164058
Fixes: elastic#163546
Fixes: elastic#162669

We tried to fix the issue with this PR
elastic#162839 but test failed again.

This is another attempt to fix it using the @jpdjere's approach where we
disable rule's table refreshing
(elastic#163698).

(cherry picked from commit 4477f64)
e40pud referenced this pull request Aug 22, 2023
…nly viewer cypress test (#164283) (#164352)

# Backport

This will backport the following commits from `main` to `8.9`:
- [[Security Solution] [Detections] Fixes flakey exceptions read-only
viewer cypress test
(#164283)](#164283)

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Ievgen
Sorokopud","email":"ievgen.sorokopud@elastic.co"},"sourceCommit":{"committedDate":"2023-08-21T18:40:14Z","message":"[Security
Solution] [Detections] Fixes flakey exceptions read-only viewer cypress
test (#164283)\n\n## Summary\r\n\r\nFixes:
https://github.com/elastic/kibana/issues/162569\r\nFixes:
https://github.com/elastic/kibana/issues/164061\r\nFixes:
https://github.com/elastic/kibana/issues/164058\r\nFixes:
https://github.com/elastic/kibana/issues/163546\r\nFixes:
https://github.com/elastic/kibana/issues/162669\r\n\r\nWe tried to fix
the issue with this PR\r\nhttps://github.com//pull/162839
but test failed again.\r\n\r\nThis is another attempt to fix it using
the @jpdjere's approach where we\r\ndisable rule's table
refreshing\r\n(https://github.com/elastic/kibana/pull/163698).","sha":"4477f642e3297355ef676dcf485efb0cb49c4fcb","branchLabelMapping":{"^v8.11.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:
SecuritySolution","Team:Detection
Engine","v8.10.0","v8.11.0"],"number":164283,"url":"https://github.com/elastic/kibana/pull/164283","mergeCommit":{"message":"[Security
Solution] [Detections] Fixes flakey exceptions read-only viewer cypress
test (#164283)\n\n## Summary\r\n\r\nFixes:
https://github.com/elastic/kibana/issues/162569\r\nFixes:
https://github.com/elastic/kibana/issues/164061\r\nFixes:
https://github.com/elastic/kibana/issues/164058\r\nFixes:
https://github.com/elastic/kibana/issues/163546\r\nFixes:
https://github.com/elastic/kibana/issues/162669\r\n\r\nWe tried to fix
the issue with this PR\r\nhttps://github.com//pull/162839
but test failed again.\r\n\r\nThis is another attempt to fix it using
the @jpdjere's approach where we\r\ndisable rule's table
refreshing\r\n(https://github.com/elastic/kibana/pull/163698).","sha":"4477f642e3297355ef676dcf485efb0cb49c4fcb"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"8.10","label":"v8.10.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/164346","number":164346,"state":"MERGED","mergeCommit":{"sha":"bdbc3e7e0e7cf59397af3653ed2f8ba86bd3ea93","message":"[8.10]
[Security Solution] [Detections] Fixes flakey exceptions read-only
viewer cypress test (#164283) (#164346)\n\n# Backport\n\nThis will
backport the following commits from `main` to `8.10`:\n- [[Security
Solution] [Detections] Fixes flakey exceptions read-only\nviewer cypress
test\n(#164283)](https://github.com/elastic/kibana/pull/164283)\n\n<!---
Backport version: 8.9.7 -->\n\n### Questions ?\nPlease refer to the
[Backport
tool\ndocumentation](https://github.com/sqren/backport)\n\n<!--BACKPORT
[{\"author\":{\"name\":\"Ievgen\nSorokopud\",\"email\":\"ievgen.sorokopud@elastic.co\"},\"sourceCommit\":{\"committedDate\":\"2023-08-21T18:40:14Z\",\"message\":\"[Security\nSolution]
[Detections] Fixes flakey exceptions read-only viewer cypress\ntest
(#164283)\\n\\n##
Summary\\r\\n\\r\\nFixes:\nhttps://github.com//issues/162569\\r\\nFixes:\nhttps://github.com//issues/164061\\r\\nFixes:\nhttps://github.com//issues/164058\\r\\nFixes:\nhttps://github.com//issues/163546\\r\\nFixes:\nhttps://github.com//issues/162669\\r\\n\\r\\nWe
tried to fix\nthe issue with this
PR\\r\\nhttps://github.com//pull/162839\nbut test failed
again.\\r\\n\\r\\nThis is another attempt to fix it using\nthe
@jpdjere's approach where we\\r\\ndisable rule's
table\nrefreshing\\r\\n(https://github.com/elastic/kibana/pull/163698).\",\"sha\":\"4477f642e3297355ef676dcf485efb0cb49c4fcb\",\"branchLabelMapping\":{\"^v8.11.0$\":\"main\",\"^v(\\\\d+).(\\\\d+).\\\\d+$\":\"$1.$2\"}},\"sourcePullRequest\":{\"labels\":[\"release_note:skip\",\"Team:\nSecuritySolution\",\"Team:Detection\nEngine\",\"v8.10.0\",\"v8.11.0\"],\"number\":164283,\"url\":\"https://github.com/elastic/kibana/pull/164283\",\"mergeCommit\":{\"message\":\"[Security\nSolution]
[Detections] Fixes flakey exceptions read-only viewer cypress\ntest
(#164283)\\n\\n##
Summary\\r\\n\\r\\nFixes:\nhttps://github.com//issues/162569\\r\\nFixes:\nhttps://github.com//issues/164061\\r\\nFixes:\nhttps://github.com//issues/164058\\r\\nFixes:\nhttps://github.com//issues/163546\\r\\nFixes:\nhttps://github.com//issues/162669\\r\\n\\r\\nWe
tried to fix\nthe issue with this
PR\\r\\nhttps://github.com//pull/162839\nbut test failed
again.\\r\\n\\r\\nThis is another attempt to fix it using\nthe
@jpdjere's approach where we\\r\\ndisable rule's
table\nrefreshing\\r\\n(https://github.com/elastic/kibana/pull/163698).\",\"sha\":\"4477f642e3297355ef676dcf485efb0cb49c4fcb\"}},\"sourceBranch\":\"main\",\"suggestedTargetBranches\":[\"8.10\"],\"targetPullRequestStates\":[{\"branch\":\"8.10\",\"label\":\"v8.10.0\",\"labelRegex\":\"^v(\\\\d+).(\\\\d+).\\\\d+$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"},{\"branch\":\"main\",\"label\":\"v8.11.0\",\"labelRegex\":\"^v8.11.0$\",\"isSourceBranch\":true,\"state\":\"MERGED\",\"url\":\"https://github.com/elastic/kibana/pull/164283\",\"number\":164283,\"mergeCommit\":{\"message\":\"[Security\nSolution]
[Detections] Fixes flakey exceptions read-only viewer cypress\ntest
(#164283)\\n\\n##
Summary\\r\\n\\r\\nFixes:\nhttps://github.com//issues/162569\\r\\nFixes:\nhttps://github.com//issues/164061\\r\\nFixes:\nhttps://github.com//issues/164058\\r\\nFixes:\nhttps://github.com//issues/163546\\r\\nFixes:\nhttps://github.com//issues/162669\\r\\n\\r\\nWe
tried to fix\nthe issue with this
PR\\r\\nhttps://github.com//pull/162839\nbut test failed
again.\\r\\n\\r\\nThis is another attempt to fix it using\nthe
@jpdjere's approach where we\\r\\ndisable rule's
table\nrefreshing\\r\\n(https://github.com/elastic/kibana/pull/163698).\",\"sha\":\"4477f642e3297355ef676dcf485efb0cb49c4fcb\"}}]}]\nBACKPORT-->\n\nCo-authored-by:
Ievgen Sorokopud
<ievgen.sorokopud@elastic.co>"}},{"branch":"main","label":"v8.11.0","labelRegex":"^v8.11.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/164283","number":164283,"mergeCommit":{"message":"[Security
Solution] [Detections] Fixes flakey exceptions read-only viewer cypress
test (#164283)\n\n## Summary\r\n\r\nFixes:
https://github.com/elastic/kibana/issues/162569\r\nFixes:
https://github.com/elastic/kibana/issues/164061\r\nFixes:
https://github.com/elastic/kibana/issues/164058\r\nFixes:
https://github.com/elastic/kibana/issues/163546\r\nFixes:
https://github.com/elastic/kibana/issues/162669\r\n\r\nWe tried to fix
the issue with this PR\r\nhttps://github.com//pull/162839
but test failed again.\r\n\r\nThis is another attempt to fix it using
the @jpdjere's approach where we\r\ndisable rule's table
refreshing\r\n(https://github.com/elastic/kibana/pull/163698).","sha":"4477f642e3297355ef676dcf485efb0cb49c4fcb"}}]}]
BACKPORT-->
bryce-b pushed a commit that referenced this pull request Aug 22, 2023
…actions.cy.ts` (#163698)

Fixes: #154721

## Summary

- Fixes flaky test:
`x-pack/plugins/security_solution/cypress/e2e/detection_response/rule_management/rule_actions/bulk_actions/bulk_edit_rules_actions.cy.ts`
- Test title: `Detection rules, bulk edit of rule actions`


## Details

For: `Detection rules, bulk edit of rule actions - Restricted action
privileges - User with no privileges can't add rule actions`
- Since this test logs in with a user with missing privileges, the
"Missing privileges" callout is shown at the top of the Rules Table. The
`selectNumberOfRules();` command selects rules one by one by clicking on
their checkboxes. However, flakiness was caused when the callout was
rendered while the selection of the rules was happening, causing a
layout shift that caused the selection of a checkbox to lose focus, and
not being able to be checked. This was solved by waiting the callout to
be rendered before the selection of rules start, with the new
`waitForCallOutToBeShown`method.

For: `Detection rules, bulk edit of rule actions - All actions
privileges - before/beforeEach Clause`
- Tests were failing in the `beforeEach` clause because the first test,
mentioned above, would be logged in with a `ROLES.hunter_no_actions`
role, and logging in with a user with permissions happened in a `before`
clause instead of a `beforeEach` clause. This caused the rest of the
suite to continue with a role without permissions, and the setup of the
second test would fail as the API requests done would fail with `401`.
Moving the initial logging-in from a `before` clause to a `beforeEach`
clause solved this issue.

For: `Detection rules, bulk edit of rule actions - All actions
privileges - Add a rule action to rules (existing connector)`
- This flakiness was extremely rare, but could be reproduced after about
400 iterations. It was caused by a similar reason as the first case
above: while rules were being selected one by one, the table would auto
refresh and focus would be lost from the checkbox that was about to be
selected. This aws fixed by disabling autorefresh in the setup.

### Other changes
- Prevents the installation of `security_detection_engine` package and
creates mock rules instead.
- Creates the `waitForCallOutToBeShown` method and moves the callout IDs
spread across different files to a a single file where they are exported
from.

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
jpdjere added a commit that referenced this pull request Aug 24, 2023
…ules_actions.cy.ts (#164569)

**NOTE: This is a manual backport of
#163698

**Original PR description:**

Fixes: #154721

## Summary

- Fixes flaky test:
`x-pack/plugins/security_solution/cypress/e2e/detection_response/rule_management/rule_actions/bulk_actions/bulk_edit_rules_actions.cy.ts`
- Test title: `Detection rules, bulk edit of rule actions`


## Details

For: `Detection rules, bulk edit of rule actions - Restricted action
privileges - User with no privileges can't add rule actions`
- Since this test logs in with a user with missing privileges, the
"Missing privileges" callout is shown at the top of the Rules Table. The
`selectNumberOfRules();` command selects rules one by one by clicking on
their checkboxes. However, flakiness was caused when the callout was
rendered while the selection of the rules was happening, causing a
layout shift that caused the selection of a checkbox to lose focus, and
not being able to be checked. This was solved by waiting the callout to
be rendered before the selection of rules start, with the new
`waitForCallOutToBeShown`method.

For: `Detection rules, bulk edit of rule actions - All actions
privileges - before/beforeEach Clause`
- Tests were failing in the `beforeEach` clause because the first test,
mentioned above, would be logged in with a `ROLES.hunter_no_actions`
role, and logging in with a user with permissions happened in a `before`
clause instead of a `beforeEach` clause. This caused the rest of the
suite to continue with a role without permissions, and the setup of the
second test would fail as the API requests done would fail with `401`.
Moving the initial logging-in from a `before` clause to a `beforeEach`
clause solved this issue.

For: `Detection rules, bulk edit of rule actions - All actions
privileges - Add a rule action to rules (existing connector)`
- This flakiness was extremely rare, but could be reproduced after about
400 iterations. It was caused by a similar reason as the first case
above: while rules were being selected one by one, the table would auto
refresh and focus would be lost from the checkbox that was about to be
selected. This aws fixed by disabling autorefresh in the setup.

### Other changes
- Prevents the installation of `security_detection_engine` package and
creates mock rules instead.
- Creates the `waitForCallOutToBeShown` method and moves the callout IDs
spread across different files to a a single file where they are exported
from.
@jpdjere jpdjere added the flake-docs Temp label to gather PRs used to create dev docs label Aug 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport:skip This PR does not require backporting flake-docs Temp label to gather PRs used to create dev docs release_note:skip Skip the PR/issue when compiling release notes Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. test v8.10.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Security Solution] Unskip flaky Cypress tests for bulk editing rule actions

7 participants