Skip to content

[6.x] Stabilize and bring back flaky tests (#13034)#13159

Merged
stacey-gammon merged 1 commit intoelastic:6.xfrom
stacey-gammon:6.x/test/stabilize-tests
Jul 28, 2017
Merged

[6.x] Stabilize and bring back flaky tests (#13034)#13159
stacey-gammon merged 1 commit intoelastic:6.xfrom
stacey-gammon:6.x/test/stabilize-tests

Conversation

@stacey-gammon
Copy link
Copy Markdown

@stacey-gammon stacey-gammon commented Jul 27, 2017

  • Stabilize tests and bring back flaky test

  • Try to fix flaky map radius visualize test

  • fix flaky tile map zoom test

  • Try to fix flaky zoom test by avoiding zooming in and out with spy panel open

  • Remove duplicate wrapped retry around find methods

The retry was moved internally, around a check for stale elements as
well, so this should no longer be neccessary.

backports #13034

Stabilize and bring back flaky tests (elastic#13034)
6578e78 
* Stabalize tests and bring back flaky test

* Try to fix flaky map radius visualize test

* fix flaky tile map zoom test

* Try to fix flaky zoom test by avoiding zooming in and out with spy panel open

* Remove duplicate wrapped retry around find methods

The retry was moved internally, around a check for stale elements as
well, so this should no longer be neccessary.
@stacey-gammon stacey-gammon added the backport This PR is a backport of another PR label Jul 27, 2017
@stacey-gammon
Copy link
Copy Markdown
Author

stacey-gammon commented Jul 27, 2017

Oh no a failure! 😢

 └-> doc view should show oldest time first
08:00:37.893          └-> "before each" hook: global before each
08:00:37.901            │ debg  Kibana uiSettings are in elasticsearch and the server is reporting a green status
08:00:38.005          │ debg  Taking screenshot "/var/lib/jenkins/workspace/elastic+kibana+pull-request+multijob-selenium/test/functional/screenshots/failure/discover app discover app field data doc view should show oldest time first.png"
08:00:38.265        └- ✖ fail: "discover app discover app field data doc view should show oldest time first"
08:00:38.265        │        Error: expected 'September 22nd 2015, 23:50:13.253\ntype:apache index:logstash-2015.09.22 @timestamp:September 22nd 2015, 23:50:13.253 ip:238.171.34.42 extension:jpg response:200 geo.coordinates:{ "lat": 38.66494528, "lon": -88.45299556 } geo.src:FR geo.dest:KH geo.srcdest:FR:KH @tags:success, info utc_time:September 22nd 2015, 23:50:13.253 referer:http://twitter.com/success/nancy-currie agent:Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322) clientip:238.171.34.42 bytes:7,124 host:media-for-the-masses.theacademyofperformingartsandscience.org request:/uploads/karl-henize.jpg url:https://media-for-the-masses.theacademyofperformingartsandscience.org/uploads/karl-henize.jpg @message:238.171.34.42 - - [2015-09-22T23:50:13.253Z] "GET /uploads/karl-henize.jpg HTTP/1.1" 200 7124 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)" spaces:this is a thing with lots of spaces wwwwoooooo xss:<script>console.log("xss")</script> headings:<h3>alexander-viktorenko</h5>, http://nytimes.com/warning/michael-massimino links:@www.slate.com, http://www.slate.com/security/frederick-w-leslie, www.www.slate.com relatedContent:{ "url": "http://www.laweekly.com/music/bjork-at-the-nokia-theatre-12-12-2408191", "og:type": "article", "og:title": "Bjork at the Nokia Theatre, 12/12", "og:description": "Bjork at the Nokia Theater, December 12 By Randall Roberts Last night&rsquo;s Bjork show at the Dystopia &ndash; er, I mean Nokia -- Theatre downtown di...", "og:url": "http://www.laweekly.com/music/bjork-at-the-nokia-theatre-12-12-2408191", "article:published_time": "2007-12-13T12:19:35-08:00", "article:modified_time": "2014-11-27T08:28:42-08:00", "article:section": "Music", "og:image": "http://IMAGES1.laweekly.com/imager/bjork-at-the-nokia-theatre-12-12/u/original/2470701/bjorktn003.jpg", "og:image:height": "334", "og:image:width": "480", "og:site_name": "LA Weekly", "twitter:title": "Bjork at the Nokia Theatre, 12/12", "twitter:description": "Bjork at the Nokia Theater, December 12 By Randall Roberts Last night&rsquo;s Bjork show at the Dystopia &ndash; er, I mean Nokia -- Theatre downtown di...", "twitter:card": "summary", "twitter:image": "http://IMAGES1.laweekly.com/imager/bjork-at-the-nokia-theatre-12-12/u/original/2470701/bjorktn003.jpg", "twitter:site": "@laweekly" }, { "url": "http://www.laweekly.com/music/the-rapture-at-the-mayan-7-25-2401011", "og:type": "article", "og:title": "The Rapture at the Mayan, 7/25", "og:description": "If you haven&rsquo;t yet experienced the phenomenon of people walk-dancing, apparently the best place to witness this is at a Rapture show. Here&rsquo;s...", "og:url": "http://www.laweekly.com/music/the-rapture-at-the-mayan-7-25-2401011", "article:published_time": "2007-07-26T12:42:30-07:00", "article:modified_time": "2014-11-27T08:00:51-08:00", "article:section": "Music", "og:image": "http://IMAGES1.laweekly.com/imager/the-rapture-at-the-mayan-7-25/u/original/2463272/rapturetn05.jpg", "og:image:height": "321", "og:image:width": "480", "og:site_name": "LA Weekly", "twitter:title": "The Rapture at the Mayan, 7/25", "twitter:description": "If you haven&rsquo;t yet experienced the phenomenon of people walk-dancing, apparently the best place to witness this is at a Rapture show. Here&rsquo;s...", "twitter:card": "summary", "twitter:image": "http://IMAGES1.laweekly.com/imager/the-rapture-at-the-mayan-7-25/u/original/2463272/rapturetn05.jpg", "twitter:site": "@laweekly" } machine.os:win 7 machine.ram:7,516,192,768 _id:AU_x3_g4GFA8no6QjkYX _type:doc _index:logstash-2015.09.22 _score: - relatedContent.article:modified_time:November 26th 2014, 03:39:26.000, November 26th 2014, 04:36:10.000 relatedContent.article:published_time:October 17th 2005, 07:10:18.000, October 17th 2005, 07:10:18.000' to equal 'September 22nd 2015, 23:50:13.253\ntype:apache index:logstash-2015.09.22 @timestamp:September 22nd 2015, 23:50:13.253 ip:238.171.34.42 extension:jpg response:200 geo.coordinates:{ "lat": 38.66494528, "lon": -88.45299556 } geo.src:FR geo.dest:KH geo.srcdest:FR:KH @tags:success, info utc_time:September 22nd 2015, 23:50:13.253 referer:http://twitter.com/success/nancy-currie agent:Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322) clientip:238.171.34.42 bytes:7,124 host:media-for-the-masses.theacademyofperformingartsandscience.org request:/uploads/karl-henize.jpg url:https://media-for-the-masses.theacademyofperformingartsandscience.org/uploads/karl-henize.jpg @message:238.171.34.42 - - [2015-09-22T23:50:13.253Z] "GET /uploads/karl-henize.jpg HTTP/1.1" 200 7124 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)" spaces:this is a thing with lots of spaces wwwwoooooo xss:<script>console.log("xss")</script> headings:<h3>alexander-viktorenko</h5>, http://nytimes.com/warning/michael-massimino links:@www.slate.com, http://www.slate.com/security/frederick-w-leslie, www.www.slate.com relatedContent:{ "url": "http://www.laweekly.com/music/bjork-at-the-nokia-theatre-12-12-2408191", "og:type": "article", "og:title": "Bjork at the Nokia Theatre, 12/12", "og:description": "Bjork at the Nokia Theater, December 12 By Randall Roberts Last night&rsquo;s Bjork show at the Dystopia &ndash; er, I mean Nokia -- Theatre downtown di...", "og:url": "http://www.laweekly.com/music/bjork-at-the-nokia-theatre-12-12-2408191", "article:published_time": "2007-12-13T12:19:35-08:00", "article:modified_time": "2014-11-27T08:28:42-08:00", "article:section": "Music", "og:image": "http://IMAGES1.laweekly.com/imager/bjork-at-the-nokia-theatre-12-12/u/original/2470701/bjorktn003.jpg", "og:image:height": "334", "og:image:width": "480", "og:site_name": "LA Weekly", "twitter:title": "Bjork at the Nokia Theatre, 12/12", "twitter:description": "Bjork at the Nokia Theater, December 12 By Randall Roberts Last night&rsquo;s Bjork show at the Dystopia &ndash; er, I mean Nokia -- Theatre downtown di...", "twitter:card": "summary", "twitter:image": "http://IMAGES1.laweekly.com/imager/bjork-at-the-nokia-theatre-12-12/u/original/2470701/bjorktn003.jpg", "twitter:site": "@laweekly" }, { "url": "http://www.laweekly.com/music/the-rapture-at-the-mayan-7-25-2401011", "og:type": "article", "og:title": "The Rapture at the Mayan, 7/25", "og:description": "If you haven&rsquo;t yet experienced the phenomenon of people walk-dancing, apparently the best place to witness this is at a Rapture show. Here&rsquo;s...", "og:url": "http://www.laweekly.com/music/the-rapture-at-the-mayan-7-25-2401011", "article:published_time": "2007-07-26T12:42:30-07:00", "article:modified_time": "2014-11-27T08:00:51-08:00", "article:section": "Music", "og:image": "http://IMAGES1.laweekly.com/imager/the-rapture-at-the-mayan-7-25/u/original/2463272/rapturetn05.jpg", "og:image:height": "321", "og:image:width": "480", "og:site_name": "LA Weekly", "twitter:title": "The Rapture at the Mayan, 7/25", "twitter:description": "If you haven&rsquo;t yet experienced the phenomenon of people walk-dancing, apparently the best place to witness this is at a Rapture show. Here&rsquo;s...", "twitter:card": "summary", "twitter:image": "http://IMAGES1.laweekly.com/imager/the-rapture-at-the-mayan-7-25/u/original/2463272/rapturetn05.jpg", "twitter:site": "@laweekly" } machine.os:win 7 machine.ram:7,516,192,768 _id:AU_x3_g4GFA8no6QjkYX _type:doc _index:logstash-2015.09.22 _score: - relatedContent.article:modified_time:November 27th 2014, 16:00:51.000, November 27th 2014, 16:28:42.000 relatedContent.article:published_time:July 26th 2007, 19:42:30.000, December 13th 2007, 20:19:35.000'
08:00:38.265        │         at Assertion.assert (node_modules/expect.js/index.js:96:13)

Tests passed locally on 6.x... not good, not good. @elastic/kibana-discovery - you guys haven't happened to come across this kind of flaky test before, eh?

@stacey-gammon
Copy link
Copy Markdown
Author

stacey-gammon commented Jul 27, 2017

Well, lets see if this is flaky, or a legit stable error.

jenkins, test this.

@stacey-gammon stacey-gammon mentioned this pull request Jul 27, 2017
Closed
@stacey-gammon
Copy link
Copy Markdown
Author

stacey-gammon commented Jul 27, 2017

Passed the second time

Jenkins, test this

@stacey-gammon
Copy link
Copy Markdown
Author

stacey-gammon commented Jul 28, 2017

Passed twice in a row

Jenkins test this

@weltenwort
Copy link
Copy Markdown
Member

weltenwort commented Jul 28, 2017

That failure is confusing, because the document ids are actually identical with only the related_content timestamps being different. Given that they are loaded from the fixtures I don't know how that could happen. 😕

@stacey-gammon
Copy link
Copy Markdown
Author

stacey-gammon commented Jul 28, 2017

Agree @weltenwort, very strange. I filed an issue for it (#13163), if it continues to be flaky we can skip it until we can figure it out, but since it seems pretty rare, I'm inclined to leave it be for now.

@stacey-gammon stacey-gammon merged commit 67de869 into elastic:6.x Jul 28, 2017
@stacey-gammon stacey-gammon deleted the 6.x/test/stabilize-tests branch October 24, 2017 13:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

backport This PR is a backport of another PR

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@stacey-gammon @weltenwort