Increase alerting test stability and reduce flakiness#50246
Merged
mikecote merged 12 commits intoelastic:masterfrom Nov 18, 2019
Merged
Increase alerting test stability and reduce flakiness#50246mikecote merged 12 commits intoelastic:masterfrom
mikecote merged 12 commits intoelastic:masterfrom
Conversation
Contributor
|
Pinging @elastic/kibana-stack-services (Team:Stack Services) |
Contributor
Author
|
@elasticmachine merge upstream |
This comment has been minimized.
This comment has been minimized.
…into alerting/test-stability
bmcconaghy
approved these changes
Nov 14, 2019
Contributor
bmcconaghy
left a comment
There was a problem hiding this comment.
This looks reasonable to me. Thanks for prioritizing re-enabling these tests!
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Contributor
💚 Build Succeeded |
Contributor
Author
|
@elasticmachine merge upstream |
Contributor
Author
|
Changed the logic for alert tests to be a bit more consistent. Where it makes sense, the tests will use the following pattern:
|
Contributor
💚 Build Succeeded |
pmuellr
approved these changes
Nov 18, 2019
Contributor
pmuellr
left a comment
There was a problem hiding this comment.
LGTM, nice to see the CI passed!
x-pack/test/alerting_api_integration/security_and_spaces/tests/alerting/alerts.ts
Outdated
Show resolved
Hide resolved
| enabled: true, | ||
| name: 'abc', | ||
| alertTypeId: 'test.noop', | ||
| interval: '10s', |
Contributor
There was a problem hiding this comment.
I'm curious about the change in interval from 10s to 1m. Does the increased latency here help cut down on "noisy" alerts generating more test es docs?
Contributor
Author
There was a problem hiding this comment.
Yeah, I mostly bumped it for protection if ever it would execute the alert again. I haven't seen that scenario happen myself but could see it if we're waiting > 10s for es docs. There was no reason behind the 10s so I figured I might as well bump it to something less likely to run.
Contributor
💚 Build Succeeded |
Contributor
Author
|
Will merge once the flaky suite runner is finished: https://kibana-ci.elastic.co/job/kibana+flaky-test-suite-runner/10/. This will run the ciGroup1 for xpack 42 times to ensure no flaky failures. |
mikecote
added a commit
to mikecote/kibana
that referenced
this pull request
Nov 18, 2019
* Increase alerting test stability * More changes to test methodology + bug fix in throttling * Fix comments * Cleanup * Typo * Fix broken tests * Fix integration tests * Fix typo
mikecote
added a commit
that referenced
this pull request
Nov 18, 2019
jloleysens
added a commit
to jloleysens/kibana
that referenced
this pull request
Nov 18, 2019
…-fallback * 'master' of github.com:elastic/kibana: (116 commits) [Maps] move apply global filter settting from layer to source (elastic#50523) [SIEM] Fix: Empty `Source` / `Destination` shown when only ports are populated (elastic#50843) [Maps] Delay vector tile layer syncing until spritesheet is loaded (elastic#48955) [Maps] prevent users from overflowing URL when filtering by shape (elastic#50747) [DOCS] Mark Beats central management as discontinued (elastic#49423) [page_objects/common_page] convert to ts (elastic#50771) [NP Kibana Migrations ] kibana plugin home (elastic#50444) [DOCS] Shareables naming convention (elastic#50497) [ML] DF Analytics - auto-populate model_memory_limit (elastic#50714) Increase alerting test stability and reduce flakiness (elastic#50246) [ML] Remaning new_job_new folder (elastic#50917) [Telemetry] Show opt-in changes for OSS users (elastic#50831) [ML] Fix lat_long anomalies table links menu and value formatting (elastic#50916) [Dev] Fix serialising a really big string (elastic#50915) Better explanation about the Prettier recommendation (extension vs. NPM module) (elastic#50629) [Monitoring] Use a basic monitoring user for tests (elastic#47865) [Monitoring] Gracefully handle issue with filebeat indices (elastic#48929) [Monitoring] Improve permissions required around setup mode (elastic#50421) Additional validation for elasticsearch username (elastic#48247) Revert changes to use_kibana_ui_setting (elastic#50877) ... # Conflicts: # src/legacy/core_plugins/console/server/request.test.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #50078
Fixes #48709
Fixes #50074
Fixes #50079
In this PR, I'm fixing tests that started failing on master due to Task Manager changes. By being lucky, there wasn't much of a delay between scheduling a task and executing previously. With the new performance improvements and method of claiming tasks, I had to adjust our tests to not always assume tasks run fairly quickly.
Also while stabilizing the tests I found a bug with throttling which is fixed in this PR.