Add pipeline for flaky test runner job by brianseeders · Pull Request #46740 · elastic/kibana

brianseeders · 2019-09-26T20:46:38Z

How to start the job: https://kibana-ci.elastic.co/job/kibana-pipeline-flaky-test-runner/build

The job will execute the chosen test suite NUMBER_EXECUTIONS times, using 14 parallel processes on one machine. The build only happens once, then the suite is executed many times.

Duplicate test results will show up, so if the same suite errors out 5 times, all 5 instances will show up.

Example: https://kibana-ci.elastic.co/job/kibana-pipeline-flaky-test-runner/4
Example of flaky test failing multiple times: https://kibana-ci.elastic.co/job/kibana-pipeline-flaky-test-runner/4/testReport/

End of console:

11:47:39  There were 8 test suite failures.
11:47:39  [Pipeline] echo
11:47:39  The executions that failed were:
11:47:39  [Pipeline] echo
11:47:39  worker-14-0
11:47:39  worker-4-0
11:47:39  worker-4-1
11:47:39  worker-9-1
11:47:39  worker-2-1
11:47:39  worker-3-1
11:47:39  worker-13-1
11:47:39  worker-5-1
11:47:39  [Pipeline] echo
11:47:39  Please check 'Test Result' and 'Pipeline Steps' pages for more info

I was able to move a lot of Jenkinsfile code to a shared location, but it unfortunately required creating a vars directory in the root of the kibana repo. This is required by Jenkins shared libraries.

elasticmachine · 2019-09-26T22:05:01Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-09-27T16:28:03Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-09-27T18:21:41Z

Pinging @elastic/kibana-operations

elasticmachine · 2019-09-27T19:36:08Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-09-30T20:26:26Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-09-30T21:04:28Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

Jenkinsfile

brianseeders · 2019-10-01T14:43:49Z

retest

elasticmachine · 2019-10-01T16:04:47Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

brianseeders · 2019-10-01T16:19:05Z

vars/kibanaPipeline.groovy

+
+  def buildStatus = buildUtils.getBuildStatus()
+  if (buildStatus != 'SUCCESS' && buildStatus != 'ABORTED') {
+    node('flyweight') {


I did change this function a little bit so that a node isn't allocated unless an e-mail actually needs to go out... Otherwise, the code in this file is just moved over from the original Jenkinsfile

Is this something we could just run on master?

Is there actually a label set up for master? I didn't realize

elasticmachine · 2019-10-01T17:26:53Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-10-01T17:36:47Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-10-04T15:44:23Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 8a73a36

* WIP Jenkinsfile for flaky test runner * Fix syntax * A few more jenkinsfile fixes * A few more fixes * Can't round numbers in Jenkins groovy sandbox apparently * More fixes * Only do build_kbn_tp_sample_panel_action once during flaky testing * Fix path and try setting a different JOB value * Keep track of flaky test failures * Introduce flaky test for testing * Some flaky test pipeline cleanup * Fix a couple of issues with flaky test failure tracking * Update flaky test runner build name/desc with metadata * Revert "Introduce flaky test for testing" This reverts commit 202e9d86cf130ef37aff1817781a5b694aa2aa32. * Try adding a local shared library * Move local library loading logic to kibana pipeline library * Move shared groovy code to shared library * Add missed file * Add ability to specify multiple agents for flaky test runner * Update kibana-pipeline-library version * Fix bug causing early exit for oss ciGroup1

This reverts commit e894255.

… into console-token-iterator * 'console-token-iterator' of github.com:jloleysens/kibana: (184 commits) [functional/services] update webdriver lib and types (elastic#47381) Standardizing IconField implementation across the app (elastic#47196) Move ui/value_suggestions ⇒ NP data plugin (elastic#45762) Remove ui/persisted_log - Part 2 (elastic#47236) Update gulp related packages (elastic#47421) Update dependency idx to ^2.5.6 (elastic#47399) try running fewer jobs in parallel on the same worker (elastic#47403) Update webpack related packages (elastic#47402) Update jsonwebtoken related packages (elastic#47400) Update gulp related packages (major) (elastic#46665) Update dependency prettier to ^1.18.2 (elastic#47340) Update dependency @types/puppeteer to ^1.20.1 (elastic#47339) Update dependency @elastic/elasticsearch to ^7.4.0 (elastic#47338) Update dependency tar-fs to ^1.16.3 (elastic#47341) [Code] Code Integrator Component (elastic#47180) [Canvas][i18n] Sidebar (elastic#46090) Generate uuid in task Manager as Kibana uuid may not yet have been initialised [Code] Embedded Code Snippet Component (elastic#47183) Revert "Add pipeline for flaky test runner job (elastic#46740)" SearchSource: fix docvalue_fields and fields intersection logic (elastic#46724) ...

This reverts commit 7d96a13. Also reconcile changes to Jenkinsfile since original revert happened

* Revert "Revert "Add pipeline for flaky test runner job (#46740)"" This reverts commit 7d96a13. Also reconcile changes to Jenkinsfile since original revert happened * Fix param parsing and add missed change * Add missing variable

* Revert "Revert "Add pipeline for flaky test runner job (elastic#46740)"" This reverts commit 7d96a13. Also reconcile changes to Jenkinsfile since original revert happened * Fix param parsing and add missed change * Add missing variable

* Revert "Revert "Add pipeline for flaky test runner job (#46740)"" This reverts commit 7d96a13. Also reconcile changes to Jenkinsfile since original revert happened * Fix param parsing and add missed change * Add missing variable

* Revert "Revert "Add pipeline for flaky test runner job (elastic#46740)"" This reverts commit 7d96a13. Also reconcile changes to Jenkinsfile since original revert happened * Fix param parsing and add missed change * Add missing variable

* Re-add pipeline for flaky test runner job (#48781) * Revert "Revert "Add pipeline for flaky test runner job (#46740)"" This reverts commit 7d96a13. Also reconcile changes to Jenkinsfile since original revert happened * Fix param parsing and add missed change * Add missing variable * Retry git clone up to 8 times before failing a build (#50734) * Flaky test runner improvements/fixes (#49367) * Allocate multiple flaky test runner agents as executions grow * WIP support for deleting kibana install dir during CI * Add setup script for testing scripts * Add REMOVE_KIBANA_INSTALL_DIR=1 to flaky test runner * Change flaky test runner worker processes from 8 to 12 * Add labels to shell scripts in Jenkins (#49657) * Change pipeline timeout from 3 hours to 2 hours (#51098) * Remove flaky pipeline step not relevant for 6.8 * Re-add failed_tests cli * Fix PARALLEL_PIPELINE_WORKER_INDEX var

brianseeders added v8.0.0 Feature:CI Continuous integration labels Sep 26, 2019

This comment has been minimized.

Sign in to view

brianseeders force-pushed the flaky-test-runner branch from 5b4e38e to b86426a Compare September 27, 2019 18:14

brianseeders added v6.8.4 v7.3.2 v7.4.0 v7.5.0 Team:Operations Kibana-Operations Team labels Sep 27, 2019

brianseeders requested review from a team, spalger and wayneseymour September 27, 2019 18:21

brianseeders force-pushed the flaky-test-runner branch from b86426a to c0e9c28 Compare September 30, 2019 19:02

brianseeders force-pushed the flaky-test-runner branch from c0e9c28 to 903c898 Compare September 30, 2019 21:03

brianseeders commented Sep 30, 2019

View reviewed changes

Jenkinsfile Outdated Show resolved Hide resolved

snide removed the v7.4.0 label Oct 1, 2019

brianseeders force-pushed the flaky-test-runner branch from 903c898 to 65d26a8 Compare October 1, 2019 16:17

brianseeders commented Oct 1, 2019

View reviewed changes

brianseeders marked this pull request as ready for review October 1, 2019 16:19

brianseeders added the release_note:skip Skip the PR/issue when compiling release notes label Oct 1, 2019

brianseeders merged commit e894255 into elastic:master Oct 4, 2019

brianseeders deleted the flaky-test-runner branch October 4, 2019 17:53

brianseeders mentioned this pull request Oct 4, 2019

[7.x] Add pipeline for flaky test runner job (#46740) #47365

Closed

brianseeders mentioned this pull request Oct 4, 2019

[7.4] Add pipeline for flaky test runner job (#46740) #47366

Closed

brianseeders mentioned this pull request Oct 4, 2019

[7.3] Add pipeline for flaky test runner job (#46740) #47367

Closed

brianseeders mentioned this pull request Oct 4, 2019

[6.8] Add pipeline for flaky test runner job (#46740) #47368

Closed

brianseeders added a commit that referenced this pull request Oct 4, 2019

Revert "Add pipeline for flaky test runner job (#46740)"

7d96a13

This reverts commit e894255.

brianseeders mentioned this pull request Oct 21, 2019

Re-add pipeline for flaky test runner job #48781

Merged

brianseeders added a commit to brianseeders/kibana that referenced this pull request Oct 22, 2019

Revert "Revert "Add pipeline for flaky test runner job (elastic#46740)""

33623ad

This reverts commit 7d96a13. Also reconcile changes to Jenkinsfile since original revert happened

Conversation

brianseeders commented Sep 26, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticmachine commented Sep 26, 2019

💚 Build Succeeded

Uh oh!

elasticmachine commented Sep 27, 2019

💚 Build Succeeded

Uh oh!

This comment has been minimized.

elasticmachine commented Sep 27, 2019

Uh oh!

elasticmachine commented Sep 27, 2019

💚 Build Succeeded

Uh oh!

elasticmachine commented Sep 30, 2019

💔 Build Failed

Uh oh!

elasticmachine commented Sep 30, 2019

💔 Build Failed

Uh oh!

Uh oh!

brianseeders commented Oct 1, 2019

Uh oh!

elasticmachine commented Oct 1, 2019

💚 Build Succeeded

Uh oh!

brianseeders Oct 1, 2019

Choose a reason for hiding this comment

Uh oh!

spalger Oct 3, 2019

Choose a reason for hiding this comment

Uh oh!

brianseeders Oct 3, 2019

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Oct 1, 2019

💚 Build Succeeded

Uh oh!

elasticmachine commented Oct 1, 2019

💚 Build Succeeded

Uh oh!

elasticmachine commented Oct 4, 2019

💚 Build Succeeded

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

brianseeders commented Sep 26, 2019 •

edited

Loading