Skip to content

Flaky test runner improvements/fixes#49367

Merged
brianseeders merged 7 commits intoelastic:masterfrom
brianseeders:flaky-test-runner-improvements
Oct 30, 2019
Merged

Flaky test runner improvements/fixes#49367
brianseeders merged 7 commits intoelastic:masterfrom
brianseeders:flaky-test-runner-improvements

Conversation

@brianseeders
Copy link
Copy Markdown
Contributor

@brianseeders brianseeders commented Oct 25, 2019

  • Scale agents up every 24 executions, up to 3 agents max
  • Remove kibana install dir at the end of each execution to free up space
  • Move a lot of duplicated setup code from all test runner scripts to test/scripts/jenkins_test_setup.sh

Removing kibana install dirs is pretty much required for the flaky test runner to work correctly, now that we're mounting the workspace in memory

@brianseeders brianseeders added Team:Operations Kibana-Operations Team v8.0.0 release_note:skip Skip the PR/issue when compiling release notes Feature:CI Continuous integration v7.5.0 v7.6.0 v6.8.4 v7.4.2 labels Oct 25, 2019
@brianseeders brianseeders requested review from a team and spalger October 25, 2019 17:18
@brianseeders brianseeders self-assigned this Oct 25, 2019
@elasticmachine
Copy link
Copy Markdown
Contributor

Pinging @elastic/kibana-operations (Team:Operations)

Copy link
Copy Markdown
Contributor

@wayneseymour wayneseymour left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome to remove more duplication!

@elasticmachine
Copy link
Copy Markdown
Contributor

💚 Build Succeeded

@brianseeders
Copy link
Copy Markdown
Contributor Author

@elasticmachine merge upstream

@elasticmachine
Copy link
Copy Markdown
Contributor

💚 Build Succeeded

@elasticmachine
Copy link
Copy Markdown
Contributor

💚 Build Succeeded

@brianseeders
Copy link
Copy Markdown
Contributor Author

@elasticmachine merge upstream

@elasticmachine
Copy link
Copy Markdown
Contributor

💚 Build Succeeded

for(def agentNumber = 1; agentNumber <= agentCount; agentNumber++) {
for(def agentNumber = 1; agentNumber <= AGENT_COUNT; agentNumber++) {
def agentNumberInside = agentNumber
def agentExecutions = floor(EXECUTIONS/AGENT_COUNT) + (agentNumber <= EXECUTIONS%AGENT_COUNT ? 1 : 0)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this be better as a closure or method with a name?
No big deal, but names are quicker to grok, vs not being able to be lazy and having to read the code lol ahahahaa

Copy link
Copy Markdown
Contributor

@spalger spalger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@brianseeders brianseeders merged commit 6468e79 into elastic:master Oct 30, 2019
@brianseeders brianseeders deleted the flaky-test-runner-improvements branch October 30, 2019 21:24
brianseeders added a commit to brianseeders/kibana that referenced this pull request Nov 26, 2019
* Allocate multiple flaky test runner agents as executions grow

* WIP support for deleting kibana install dir during CI

* Add setup script for testing scripts

* Add REMOVE_KIBANA_INSTALL_DIR=1 to flaky test runner

* Change flaky test runner worker processes from 8 to 12
brianseeders added a commit that referenced this pull request Dec 2, 2019
* Re-add pipeline for flaky test runner job (#48781)

* Revert "Revert "Add pipeline for flaky test runner job (#46740)""

This reverts commit 7d96a13.

Also reconcile changes to Jenkinsfile since original revert happened

* Fix param parsing and add missed change

* Add missing variable

* Retry git clone up to 8 times before failing a build (#50734)

* Flaky test runner improvements/fixes (#49367)

* Allocate multiple flaky test runner agents as executions grow

* WIP support for deleting kibana install dir during CI

* Add setup script for testing scripts

* Add REMOVE_KIBANA_INSTALL_DIR=1 to flaky test runner

* Change flaky test runner worker processes from 8 to 12

* Add labels to shell scripts in Jenkins (#49657)

* Change pipeline timeout from 3 hours to 2 hours (#51098)

* Remove flaky pipeline step not relevant for 6.8

* Re-add failed_tests cli

* Fix PARALLEL_PIPELINE_WORKER_INDEX var
brianseeders added a commit to brianseeders/kibana that referenced this pull request Jan 13, 2020
* Allocate multiple flaky test runner agents as executions grow

* WIP support for deleting kibana install dir during CI

* Add setup script for testing scripts

* Add REMOVE_KIBANA_INSTALL_DIR=1 to flaky test runner

* Change flaky test runner worker processes from 8 to 12
brianseeders added a commit to brianseeders/kibana that referenced this pull request Jan 13, 2020
* Allocate multiple flaky test runner agents as executions grow

* WIP support for deleting kibana install dir during CI

* Add setup script for testing scripts

* Add REMOVE_KIBANA_INSTALL_DIR=1 to flaky test runner

* Change flaky test runner worker processes from 8 to 12
brianseeders added a commit that referenced this pull request Jan 13, 2020
* Allocate multiple flaky test runner agents as executions grow

* WIP support for deleting kibana install dir during CI

* Add setup script for testing scripts

* Add REMOVE_KIBANA_INSTALL_DIR=1 to flaky test runner

* Change flaky test runner worker processes from 8 to 12
brianseeders added a commit that referenced this pull request Jan 13, 2020
* Allocate multiple flaky test runner agents as executions grow

* WIP support for deleting kibana install dir during CI

* Add setup script for testing scripts

* Add REMOVE_KIBANA_INSTALL_DIR=1 to flaky test runner

* Change flaky test runner worker processes from 8 to 12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Feature:CI Continuous integration release_note:skip Skip the PR/issue when compiling release notes Team:Operations Kibana-Operations Team v6.8.4 v7.4.2 v7.5.0 v7.6.0 v8.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants