Skip to content

[8.9] [Security Solution] Fix flaky test: detection_rules/bulk_edit_rules_actions.cy.ts#164569

Merged
jpdjere merged 2 commits intoelastic:8.9from
jpdjere:backport-8.9-163698
Aug 24, 2023
Merged

[8.9] [Security Solution] Fix flaky test: detection_rules/bulk_edit_rules_actions.cy.ts#164569
jpdjere merged 2 commits intoelastic:8.9from
jpdjere:backport-8.9-163698

Conversation

@jpdjere
Copy link
Copy Markdown
Contributor

@jpdjere jpdjere commented Aug 23, 2023

NOTE: This is a manual backport of #163698

Original PR description:

Fixes: #154721

Summary

  • Fixes flaky test: x-pack/plugins/security_solution/cypress/e2e/detection_response/rule_management/rule_actions/bulk_actions/bulk_edit_rules_actions.cy.ts
  • Test title: Detection rules, bulk edit of rule actions

Details

For: Detection rules, bulk edit of rule actions - Restricted action privileges - User with no privileges can't add rule actions

  • Since this test logs in with a user with missing privileges, the "Missing privileges" callout is shown at the top of the Rules Table. The selectNumberOfRules(); command selects rules one by one by clicking on their checkboxes. However, flakiness was caused when the callout was rendered while the selection of the rules was happening, causing a layout shift that caused the selection of a checkbox to lose focus, and not being able to be checked. This was solved by waiting the callout to be rendered before the selection of rules start, with the new waitForCallOutToBeShownmethod.

For: Detection rules, bulk edit of rule actions - All actions privileges - before/beforeEach Clause

  • Tests were failing in the beforeEach clause because the first test, mentioned above, would be logged in with a ROLES.hunter_no_actions role, and logging in with a user with permissions happened in a before clause instead of a beforeEach clause. This caused the rest of the suite to continue with a role without permissions, and the setup of the second test would fail as the API requests done would fail with 401. Moving the initial logging-in from a before clause to a beforeEach clause solved this issue.

For: Detection rules, bulk edit of rule actions - All actions privileges - Add a rule action to rules (existing connector)

  • This flakiness was extremely rare, but could be reproduced after about 400 iterations. It was caused by a similar reason as the first case above: while rules were being selected one by one, the table would auto refresh and focus would be lost from the checkbox that was about to be selected. This aws fixed by disabling autorefresh in the setup.

Other changes

  • Prevents the installation of security_detection_engine package and creates mock rules instead.
  • Creates the waitForCallOutToBeShown method and moves the callout IDs spread across different files to a a single file where they are exported from.

@jpdjere jpdjere added test backport This PR is a backport of another PR release_note:skip Skip the PR/issue when compiling release notes Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Detection Rule Management Security Detection Rule Management Team labels Aug 23, 2023
@jpdjere jpdjere requested a review from maximpn August 23, 2023 11:49
@jpdjere jpdjere self-assigned this Aug 23, 2023
@elasticmachine
Copy link
Copy Markdown
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@elasticmachine
Copy link
Copy Markdown
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@kibana-ci
Copy link
Copy Markdown

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

Unknown metric groups

ESLint disabled line counts

id before after diff
enterpriseSearch 13 15 +2
securitySolution 416 420 +4
total +6

Total ESLint disabled count

id before after diff
enterpriseSearch 14 16 +2
securitySolution 497 501 +4
total +6

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @jpdjere

@jpdjere jpdjere merged commit 9934478 into elastic:8.9 Aug 24, 2023
@jpdjere jpdjere added the flake-docs Temp label to gather PRs used to create dev docs label Aug 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport This PR is a backport of another PR flake-docs Temp label to gather PRs used to create dev docs release_note:skip Skip the PR/issue when compiling release notes Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. test

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants