[Security Solution] Fix flaky test: detection_rules/bulk_edit_rules_actions.cy.ts#163698
Merged
jpdjere merged 11 commits intoelastic:mainfrom Aug 16, 2023
Merged
Conversation
Contributor
Author
Contributor
Author
|
Flaky test run - After fix: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2855 |
Contributor
Author
e82b22f to
5084e3c
Compare
Contributor
Author
|
[3rd change] Fix flaky test: detection_rules/bulk_edit_rules_actions.cy.ts #163698 |
Contributor
Author
|
[Fix: Disable autorefresh] Fix flaky test: detection_rules/bulk_edit_rules_actions.cy.ts #163698 |
Contributor
|
Pinging @elastic/security-detections-response (Team:Detections and Resp) |
Contributor
|
Pinging @elastic/security-solution (Team: SecuritySolution) |
e40pud
approved these changes
Aug 15, 2023
fbe0a54 to
ac09363
Compare
Contributor
|
@elasticmachine merge upstream |
💚 Build Succeeded
Metrics [docs]
History
To update your PR or re-run it, just comment with: cc @jpdjere |
hop-dev
pushed a commit
to hop-dev/kibana
that referenced
this pull request
Aug 16, 2023
…actions.cy.ts` (elastic#163698) Fixes: elastic#154721 ## Summary - Fixes flaky test: `x-pack/plugins/security_solution/cypress/e2e/detection_response/rule_management/rule_actions/bulk_actions/bulk_edit_rules_actions.cy.ts` - Test title: `Detection rules, bulk edit of rule actions` ## Details For: `Detection rules, bulk edit of rule actions - Restricted action privileges - User with no privileges can't add rule actions` - Since this test logs in with a user with missing privileges, the "Missing privileges" callout is shown at the top of the Rules Table. The `selectNumberOfRules();` command selects rules one by one by clicking on their checkboxes. However, flakiness was caused when the callout was rendered while the selection of the rules was happening, causing a layout shift that caused the selection of a checkbox to lose focus, and not being able to be checked. This was solved by waiting the callout to be rendered before the selection of rules start, with the new `waitForCallOutToBeShown`method. For: `Detection rules, bulk edit of rule actions - All actions privileges - before/beforeEach Clause` - Tests were failing in the `beforeEach` clause because the first test, mentioned above, would be logged in with a `ROLES.hunter_no_actions` role, and logging in with a user with permissions happened in a `before` clause instead of a `beforeEach` clause. This caused the rest of the suite to continue with a role without permissions, and the setup of the second test would fail as the API requests done would fail with `401`. Moving the initial logging-in from a `before` clause to a `beforeEach` clause solved this issue. For: `Detection rules, bulk edit of rule actions - All actions privileges - Add a rule action to rules (existing connector)` - This flakiness was extremely rare, but could be reproduced after about 400 iterations. It was caused by a similar reason as the first case above: while rules were being selected one by one, the table would auto refresh and focus would be lost from the checkbox that was about to be selected. This aws fixed by disabling autorefresh in the setup. ### Other changes - Prevents the installation of `security_detection_engine` package and creates mock rules instead. - Creates the `waitForCallOutToBeShown` method and moves the callout IDs spread across different files to a a single file where they are exported from. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
e40pud
added a commit
that referenced
this pull request
Aug 21, 2023
…ewer cypress test (#164283) ## Summary Fixes: #162569 Fixes: #164061 Fixes: #164058 Fixes: #163546 Fixes: #162669 We tried to fix the issue with this PR #162839 but test failed again. This is another attempt to fix it using the @jpdjere's approach where we disable rule's table refreshing (#163698).
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Aug 21, 2023
…ewer cypress test (elastic#164283) ## Summary Fixes: elastic#162569 Fixes: elastic#164061 Fixes: elastic#164058 Fixes: elastic#163546 Fixes: elastic#162669 We tried to fix the issue with this PR elastic#162839 but test failed again. This is another attempt to fix it using the @jpdjere's approach where we disable rule's table refreshing (elastic#163698). (cherry picked from commit 4477f64)
kibanamachine
referenced
this pull request
Aug 21, 2023
…only viewer cypress test (#164283) (#164346) # Backport This will backport the following commits from `main` to `8.10`: - [[Security Solution] [Detections] Fixes flakey exceptions read-only viewer cypress test (#164283)](#164283) <!--- Backport version: 8.9.7 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Ievgen Sorokopud","email":"ievgen.sorokopud@elastic.co"},"sourceCommit":{"committedDate":"2023-08-21T18:40:14Z","message":"[Security Solution] [Detections] Fixes flakey exceptions read-only viewer cypress test (#164283)\n\n## Summary\r\n\r\nFixes: https://github.com/elastic/kibana/issues/162569\r\nFixes: https://github.com/elastic/kibana/issues/164061\r\nFixes: https://github.com/elastic/kibana/issues/164058\r\nFixes: https://github.com/elastic/kibana/issues/163546\r\nFixes: https://github.com/elastic/kibana/issues/162669\r\n\r\nWe tried to fix the issue with this PR\r\nhttps://github.com//pull/162839 but test failed again.\r\n\r\nThis is another attempt to fix it using the @jpdjere's approach where we\r\ndisable rule's table refreshing\r\n(https://github.com/elastic/kibana/pull/163698).","sha":"4477f642e3297355ef676dcf485efb0cb49c4fcb","branchLabelMapping":{"^v8.11.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team: SecuritySolution","Team:Detection Engine","v8.10.0","v8.11.0"],"number":164283,"url":"https://github.com/elastic/kibana/pull/164283","mergeCommit":{"message":"[Security Solution] [Detections] Fixes flakey exceptions read-only viewer cypress test (#164283)\n\n## Summary\r\n\r\nFixes: https://github.com/elastic/kibana/issues/162569\r\nFixes: https://github.com/elastic/kibana/issues/164061\r\nFixes: https://github.com/elastic/kibana/issues/164058\r\nFixes: https://github.com/elastic/kibana/issues/163546\r\nFixes: https://github.com/elastic/kibana/issues/162669\r\n\r\nWe tried to fix the issue with this PR\r\nhttps://github.com//pull/162839 but test failed again.\r\n\r\nThis is another attempt to fix it using the @jpdjere's approach where we\r\ndisable rule's table refreshing\r\n(https://github.com/elastic/kibana/pull/163698).","sha":"4477f642e3297355ef676dcf485efb0cb49c4fcb"}},"sourceBranch":"main","suggestedTargetBranches":["8.10"],"targetPullRequestStates":[{"branch":"8.10","label":"v8.10.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.11.0","labelRegex":"^v8.11.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/164283","number":164283,"mergeCommit":{"message":"[Security Solution] [Detections] Fixes flakey exceptions read-only viewer cypress test (#164283)\n\n## Summary\r\n\r\nFixes: https://github.com/elastic/kibana/issues/162569\r\nFixes: https://github.com/elastic/kibana/issues/164061\r\nFixes: https://github.com/elastic/kibana/issues/164058\r\nFixes: https://github.com/elastic/kibana/issues/163546\r\nFixes: https://github.com/elastic/kibana/issues/162669\r\n\r\nWe tried to fix the issue with this PR\r\nhttps://github.com//pull/162839 but test failed again.\r\n\r\nThis is another attempt to fix it using the @jpdjere's approach where we\r\ndisable rule's table refreshing\r\n(https://github.com/elastic/kibana/pull/163698).","sha":"4477f642e3297355ef676dcf485efb0cb49c4fcb"}}]}] BACKPORT--> Co-authored-by: Ievgen Sorokopud <ievgen.sorokopud@elastic.co>
e40pud
added a commit
to e40pud/kibana
that referenced
this pull request
Aug 21, 2023
…ewer cypress test (elastic#164283) ## Summary Fixes: elastic#162569 Fixes: elastic#164061 Fixes: elastic#164058 Fixes: elastic#163546 Fixes: elastic#162669 We tried to fix the issue with this PR elastic#162839 but test failed again. This is another attempt to fix it using the @jpdjere's approach where we disable rule's table refreshing (elastic#163698). (cherry picked from commit 4477f64)
e40pud
referenced
this pull request
Aug 22, 2023
…nly viewer cypress test (#164283) (#164352) # Backport This will backport the following commits from `main` to `8.9`: - [[Security Solution] [Detections] Fixes flakey exceptions read-only viewer cypress test (#164283)](#164283) <!--- Backport version: 8.9.8 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Ievgen Sorokopud","email":"ievgen.sorokopud@elastic.co"},"sourceCommit":{"committedDate":"2023-08-21T18:40:14Z","message":"[Security Solution] [Detections] Fixes flakey exceptions read-only viewer cypress test (#164283)\n\n## Summary\r\n\r\nFixes: https://github.com/elastic/kibana/issues/162569\r\nFixes: https://github.com/elastic/kibana/issues/164061\r\nFixes: https://github.com/elastic/kibana/issues/164058\r\nFixes: https://github.com/elastic/kibana/issues/163546\r\nFixes: https://github.com/elastic/kibana/issues/162669\r\n\r\nWe tried to fix the issue with this PR\r\nhttps://github.com//pull/162839 but test failed again.\r\n\r\nThis is another attempt to fix it using the @jpdjere's approach where we\r\ndisable rule's table refreshing\r\n(https://github.com/elastic/kibana/pull/163698).","sha":"4477f642e3297355ef676dcf485efb0cb49c4fcb","branchLabelMapping":{"^v8.11.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team: SecuritySolution","Team:Detection Engine","v8.10.0","v8.11.0"],"number":164283,"url":"https://github.com/elastic/kibana/pull/164283","mergeCommit":{"message":"[Security Solution] [Detections] Fixes flakey exceptions read-only viewer cypress test (#164283)\n\n## Summary\r\n\r\nFixes: https://github.com/elastic/kibana/issues/162569\r\nFixes: https://github.com/elastic/kibana/issues/164061\r\nFixes: https://github.com/elastic/kibana/issues/164058\r\nFixes: https://github.com/elastic/kibana/issues/163546\r\nFixes: https://github.com/elastic/kibana/issues/162669\r\n\r\nWe tried to fix the issue with this PR\r\nhttps://github.com//pull/162839 but test failed again.\r\n\r\nThis is another attempt to fix it using the @jpdjere's approach where we\r\ndisable rule's table refreshing\r\n(https://github.com/elastic/kibana/pull/163698).","sha":"4477f642e3297355ef676dcf485efb0cb49c4fcb"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"8.10","label":"v8.10.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/164346","number":164346,"state":"MERGED","mergeCommit":{"sha":"bdbc3e7e0e7cf59397af3653ed2f8ba86bd3ea93","message":"[8.10] [Security Solution] [Detections] Fixes flakey exceptions read-only viewer cypress test (#164283) (#164346)\n\n# Backport\n\nThis will backport the following commits from `main` to `8.10`:\n- [[Security Solution] [Detections] Fixes flakey exceptions read-only\nviewer cypress test\n(#164283)](https://github.com/elastic/kibana/pull/164283)\n\n<!--- Backport version: 8.9.7 -->\n\n### Questions ?\nPlease refer to the [Backport tool\ndocumentation](https://github.com/sqren/backport)\n\n<!--BACKPORT [{\"author\":{\"name\":\"Ievgen\nSorokopud\",\"email\":\"ievgen.sorokopud@elastic.co\"},\"sourceCommit\":{\"committedDate\":\"2023-08-21T18:40:14Z\",\"message\":\"[Security\nSolution] [Detections] Fixes flakey exceptions read-only viewer cypress\ntest (#164283)\\n\\n## Summary\\r\\n\\r\\nFixes:\nhttps://github.com//issues/162569\\r\\nFixes:\nhttps://github.com//issues/164061\\r\\nFixes:\nhttps://github.com//issues/164058\\r\\nFixes:\nhttps://github.com//issues/163546\\r\\nFixes:\nhttps://github.com//issues/162669\\r\\n\\r\\nWe tried to fix\nthe issue with this PR\\r\\nhttps://github.com//pull/162839\nbut test failed again.\\r\\n\\r\\nThis is another attempt to fix it using\nthe @jpdjere's approach where we\\r\\ndisable rule's table\nrefreshing\\r\\n(https://github.com/elastic/kibana/pull/163698).\",\"sha\":\"4477f642e3297355ef676dcf485efb0cb49c4fcb\",\"branchLabelMapping\":{\"^v8.11.0$\":\"main\",\"^v(\\\\d+).(\\\\d+).\\\\d+$\":\"$1.$2\"}},\"sourcePullRequest\":{\"labels\":[\"release_note:skip\",\"Team:\nSecuritySolution\",\"Team:Detection\nEngine\",\"v8.10.0\",\"v8.11.0\"],\"number\":164283,\"url\":\"https://github.com/elastic/kibana/pull/164283\",\"mergeCommit\":{\"message\":\"[Security\nSolution] [Detections] Fixes flakey exceptions read-only viewer cypress\ntest (#164283)\\n\\n## Summary\\r\\n\\r\\nFixes:\nhttps://github.com//issues/162569\\r\\nFixes:\nhttps://github.com//issues/164061\\r\\nFixes:\nhttps://github.com//issues/164058\\r\\nFixes:\nhttps://github.com//issues/163546\\r\\nFixes:\nhttps://github.com//issues/162669\\r\\n\\r\\nWe tried to fix\nthe issue with this PR\\r\\nhttps://github.com//pull/162839\nbut test failed again.\\r\\n\\r\\nThis is another attempt to fix it using\nthe @jpdjere's approach where we\\r\\ndisable rule's table\nrefreshing\\r\\n(https://github.com/elastic/kibana/pull/163698).\",\"sha\":\"4477f642e3297355ef676dcf485efb0cb49c4fcb\"}},\"sourceBranch\":\"main\",\"suggestedTargetBranches\":[\"8.10\"],\"targetPullRequestStates\":[{\"branch\":\"8.10\",\"label\":\"v8.10.0\",\"labelRegex\":\"^v(\\\\d+).(\\\\d+).\\\\d+$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"},{\"branch\":\"main\",\"label\":\"v8.11.0\",\"labelRegex\":\"^v8.11.0$\",\"isSourceBranch\":true,\"state\":\"MERGED\",\"url\":\"https://github.com/elastic/kibana/pull/164283\",\"number\":164283,\"mergeCommit\":{\"message\":\"[Security\nSolution] [Detections] Fixes flakey exceptions read-only viewer cypress\ntest (#164283)\\n\\n## Summary\\r\\n\\r\\nFixes:\nhttps://github.com//issues/162569\\r\\nFixes:\nhttps://github.com//issues/164061\\r\\nFixes:\nhttps://github.com//issues/164058\\r\\nFixes:\nhttps://github.com//issues/163546\\r\\nFixes:\nhttps://github.com//issues/162669\\r\\n\\r\\nWe tried to fix\nthe issue with this PR\\r\\nhttps://github.com//pull/162839\nbut test failed again.\\r\\n\\r\\nThis is another attempt to fix it using\nthe @jpdjere's approach where we\\r\\ndisable rule's table\nrefreshing\\r\\n(https://github.com/elastic/kibana/pull/163698).\",\"sha\":\"4477f642e3297355ef676dcf485efb0cb49c4fcb\"}}]}]\nBACKPORT-->\n\nCo-authored-by: Ievgen Sorokopud <ievgen.sorokopud@elastic.co>"}},{"branch":"main","label":"v8.11.0","labelRegex":"^v8.11.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/164283","number":164283,"mergeCommit":{"message":"[Security Solution] [Detections] Fixes flakey exceptions read-only viewer cypress test (#164283)\n\n## Summary\r\n\r\nFixes: https://github.com/elastic/kibana/issues/162569\r\nFixes: https://github.com/elastic/kibana/issues/164061\r\nFixes: https://github.com/elastic/kibana/issues/164058\r\nFixes: https://github.com/elastic/kibana/issues/163546\r\nFixes: https://github.com/elastic/kibana/issues/162669\r\n\r\nWe tried to fix the issue with this PR\r\nhttps://github.com//pull/162839 but test failed again.\r\n\r\nThis is another attempt to fix it using the @jpdjere's approach where we\r\ndisable rule's table refreshing\r\n(https://github.com/elastic/kibana/pull/163698).","sha":"4477f642e3297355ef676dcf485efb0cb49c4fcb"}}]}] BACKPORT-->
bryce-b
pushed a commit
that referenced
this pull request
Aug 22, 2023
…actions.cy.ts` (#163698) Fixes: #154721 ## Summary - Fixes flaky test: `x-pack/plugins/security_solution/cypress/e2e/detection_response/rule_management/rule_actions/bulk_actions/bulk_edit_rules_actions.cy.ts` - Test title: `Detection rules, bulk edit of rule actions` ## Details For: `Detection rules, bulk edit of rule actions - Restricted action privileges - User with no privileges can't add rule actions` - Since this test logs in with a user with missing privileges, the "Missing privileges" callout is shown at the top of the Rules Table. The `selectNumberOfRules();` command selects rules one by one by clicking on their checkboxes. However, flakiness was caused when the callout was rendered while the selection of the rules was happening, causing a layout shift that caused the selection of a checkbox to lose focus, and not being able to be checked. This was solved by waiting the callout to be rendered before the selection of rules start, with the new `waitForCallOutToBeShown`method. For: `Detection rules, bulk edit of rule actions - All actions privileges - before/beforeEach Clause` - Tests were failing in the `beforeEach` clause because the first test, mentioned above, would be logged in with a `ROLES.hunter_no_actions` role, and logging in with a user with permissions happened in a `before` clause instead of a `beforeEach` clause. This caused the rest of the suite to continue with a role without permissions, and the setup of the second test would fail as the API requests done would fail with `401`. Moving the initial logging-in from a `before` clause to a `beforeEach` clause solved this issue. For: `Detection rules, bulk edit of rule actions - All actions privileges - Add a rule action to rules (existing connector)` - This flakiness was extremely rare, but could be reproduced after about 400 iterations. It was caused by a similar reason as the first case above: while rules were being selected one by one, the table would auto refresh and focus would be lost from the checkbox that was about to be selected. This aws fixed by disabling autorefresh in the setup. ### Other changes - Prevents the installation of `security_detection_engine` package and creates mock rules instead. - Creates the `waitForCallOutToBeShown` method and moves the callout IDs spread across different files to a a single file where they are exported from. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
jpdjere
added a commit
that referenced
this pull request
Aug 24, 2023
…ules_actions.cy.ts (#164569) **NOTE: This is a manual backport of #163698 **Original PR description:** Fixes: #154721 ## Summary - Fixes flaky test: `x-pack/plugins/security_solution/cypress/e2e/detection_response/rule_management/rule_actions/bulk_actions/bulk_edit_rules_actions.cy.ts` - Test title: `Detection rules, bulk edit of rule actions` ## Details For: `Detection rules, bulk edit of rule actions - Restricted action privileges - User with no privileges can't add rule actions` - Since this test logs in with a user with missing privileges, the "Missing privileges" callout is shown at the top of the Rules Table. The `selectNumberOfRules();` command selects rules one by one by clicking on their checkboxes. However, flakiness was caused when the callout was rendered while the selection of the rules was happening, causing a layout shift that caused the selection of a checkbox to lose focus, and not being able to be checked. This was solved by waiting the callout to be rendered before the selection of rules start, with the new `waitForCallOutToBeShown`method. For: `Detection rules, bulk edit of rule actions - All actions privileges - before/beforeEach Clause` - Tests were failing in the `beforeEach` clause because the first test, mentioned above, would be logged in with a `ROLES.hunter_no_actions` role, and logging in with a user with permissions happened in a `before` clause instead of a `beforeEach` clause. This caused the rest of the suite to continue with a role without permissions, and the setup of the second test would fail as the API requests done would fail with `401`. Moving the initial logging-in from a `before` clause to a `beforeEach` clause solved this issue. For: `Detection rules, bulk edit of rule actions - All actions privileges - Add a rule action to rules (existing connector)` - This flakiness was extremely rare, but could be reproduced after about 400 iterations. It was caused by a similar reason as the first case above: while rules were being selected one by one, the table would auto refresh and focus would be lost from the checkbox that was about to be selected. This aws fixed by disabling autorefresh in the setup. ### Other changes - Prevents the installation of `security_detection_engine` package and creates mock rules instead. - Creates the `waitForCallOutToBeShown` method and moves the callout IDs spread across different files to a a single file where they are exported from.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes: #154721
Summary
x-pack/plugins/security_solution/cypress/e2e/detection_response/rule_management/rule_actions/bulk_actions/bulk_edit_rules_actions.cy.tsDetection rules, bulk edit of rule actionsDetails
For:
Detection rules, bulk edit of rule actions - Restricted action privileges - User with no privileges can't add rule actionsselectNumberOfRules();command selects rules one by one by clicking on their checkboxes. However, flakiness was caused when the callout was rendered while the selection of the rules was happening, causing a layout shift that caused the selection of a checkbox to lose focus, and not being able to be checked. This was solved by waiting the callout to be rendered before the selection of rules start, with the newwaitForCallOutToBeShownmethod.For:
Detection rules, bulk edit of rule actions - All actions privileges - before/beforeEach ClausebeforeEachclause because the first test, mentioned above, would be logged in with aROLES.hunter_no_actionsrole, and logging in with a user with permissions happened in abeforeclause instead of abeforeEachclause. This caused the rest of the suite to continue with a role without permissions, and the setup of the second test would fail as the API requests done would fail with401. Moving the initial logging-in from abeforeclause to abeforeEachclause solved this issue.For:
Detection rules, bulk edit of rule actions - All actions privileges - Add a rule action to rules (existing connector)Other changes
security_detection_enginepackage and creates mock rules instead.waitForCallOutToBeShownmethod and moves the callout IDs spread across different files to a a single file where they are exported from.