Skip to content

Turn on racetest for all presubmits#3780

Merged
xiaolanz merged 4 commits intoistio:masterfrom
xiaolanz:infra
Mar 19, 2018
Merged

Turn on racetest for all presubmits#3780
xiaolanz merged 4 commits intoistio:masterfrom
xiaolanz:infra

Conversation

@xiaolanz
Copy link
Copy Markdown
Contributor

@xiaolanz xiaolanz commented Feb 26, 2018

addresses #4235

We will watch submission queue for a few days before turn on "required"

@xiaolanz xiaolanz requested review from a team, GregHanson, hklai and ldemailly February 26, 2018 18:58
@istio-merge-robot istio-merge-robot added the needs-rebase Indicates a PR needs to be rebased before being merged label Feb 26, 2018
@istio-merge-robot
Copy link
Copy Markdown

@xiaolanz PR needs rebase

@istio-merge-robot istio-merge-robot removed the needs-rebase Indicates a PR needs to be rebased before being merged label Feb 26, 2018
@ldemailly
Copy link
Copy Markdown
Member

looks like there are still failures?

@xiaolanz
Copy link
Copy Markdown
Contributor Author

Looks like some tests are being flaking.

/retest

@ldemailly
Copy link
Copy Markdown
Member

circle doesn't understand "/retest" you have to click the button in the UI

@ldemailly
Copy link
Copy Markdown
Member

istio-merge-robot pushed a commit that referenced this pull request Feb 27, 2018
Automatic merge from submit-queue.

fix race test for TestWatchChanges

address: #3780

go test --v ./mixer/pkg/runtime2/... --race -run TestWatchChanges
@codecov
Copy link
Copy Markdown

codecov bot commented Mar 7, 2018

Codecov Report

Merging #3780 into master will decrease coverage by 1%.
The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff           @@
##           master   #3780    +/-   ##
=======================================
- Coverage      71%     70%   -<1%     
=======================================
  Files         317     317            
  Lines       28911   29516   +605     
=======================================
+ Hits        20247   20581   +334     
- Misses       7482    7693   +211     
- Partials     1182    1242    +60
Impacted Files Coverage Δ
mixer/adapter/solarwinds/metrics_handler.go 75% <0%> (-8%) ⬇️
mixer/adapter/kubernetesenv/cache.go 83% <0%> (-2%) ⬇️
mixer/adapter/opa/opa.go 76% <0%> (-2%) ⬇️
mixer/adapter/fluentd/fluentd.go 76% <0%> (-1%) ⬇️
mixer/adapter/kubernetesenv/kubernetesenv.go 68% <0%> (-1%) ⬇️
pilot/pkg/proxy/envoy/v2/mesh.go 75% <0%> (ø) ⬇️
mixer/template/sample/template.gen.go 55% <0%> (ø) ⬇️
mixer/adapter/circonus/circonus.go 71% <0%> (ø) ⬇️
mixer/adapter/dogstatsd/dogstatsd.go 100% <0%> (ø) ⬆️
mixer/adapter/stdio/stdio.go 100% <0%> (ø) ⬆️
... and 8 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 59764d1...211f944. Read the comment docs.

@xiaolanz xiaolanz requested a review from a team March 9, 2018 20:25
@istio-merge-robot istio-merge-robot added the needs-rebase Indicates a PR needs to be rebased before being merged label Mar 9, 2018
@istio-merge-robot
Copy link
Copy Markdown

@xiaolanz PR needs rebase

@istio-merge-robot istio-merge-robot removed the needs-rebase Indicates a PR needs to be rebased before being merged label Mar 17, 2018
@xiaolanz
Copy link
Copy Markdown
Contributor Author

PTAL

Copy link
Copy Markdown
Contributor

@kyessenov kyessenov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/approve
/lgtm

@kyessenov
Copy link
Copy Markdown
Contributor

/retest istio-unit-tests

- e2e-simple:
requires:
- build
- racetest:
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer need dependencies - but you may run it after test.

@sebastienvas
Copy link
Copy Markdown
Contributor

/approve

Copy link
Copy Markdown
Contributor

@costinm costinm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please don't add it to presubmit until it is consistently greed for few days.

It failed 3 hours ago in the flaky suite.

@istio-merge-robot
Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kyessenov, sebastienvas

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these OWNERS Files:
  • OWNERS [kyessenov,sebastienvas]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

@istio-merge-robot
Copy link
Copy Markdown

/test all [submit-queue is verifying that this PR is safe to merge]

@istio-testing
Copy link
Copy Markdown
Collaborator

New changes are detected. LGTM label has been removed.

@xiaolanz
Copy link
Copy Markdown
Contributor Author

@costinm this turns on for presubmit but doesn't block. So i can find the flakies.

@sebastienvas
Copy link
Copy Markdown
Contributor

We should NEVER move a whole suite to flaky when only one test is flaky. Instead we should create an ignore file, where you put the flaky test, and file a bug when this happen. Preventing all the tests from running is just an open doors for more coming bugs.

@sebastienvas
Copy link
Copy Markdown
Contributor

Note that you can skip a test for flaky directly in the test which is even better than the ignore file.

@costinm
Copy link
Copy Markdown
Contributor

costinm commented Mar 19, 2018

Yes - it's fine to ignore/skip tests from the suite if a bug is filled. But so far this has not been done for
racetests.

Where the test is running should have no impact on fixing the bugs - all tests should upload to the
dashboard, and failing tests need to be investigated and fixed.

The cirlceCI presubmit is intended to contain a subset of the tests suites, with <15 min run time and very reliable. There are 2 periodic runners and a nightly build, running other suites - and prow. There are cases where the presubmit will not catch bugs that are covered by longer-running suites - passsing
presubmit doesn't mean a change is perfect, or that if a bug is found in the load test or periodic suites
it is lower priority.

@xiaolanz xiaolanz merged commit af16765 into istio:master Mar 19, 2018
@sebastienvas
Copy link
Copy Markdown
Contributor

Let's discuss this elsewhere. The 15 mn requirement only makes sense when you know what are testing for. Your presubmit time should be defined by the % of postsubmit failure not just by a random number. 15mn is something we want to reach eventually, but not a hard stop.

@ldemailly
Copy link
Copy Markdown
Member

back to the topic of the PR, racetest is still failing:

for instance https://circleci.com/gh/istio/istio/52665

@xiaolanz
Copy link
Copy Markdown
Contributor Author

This is not enforced so new tests can break racetest.

TestKeyCertBundleRotator is a regression. Filed bug #4385

@ldemailly
Copy link
Copy Markdown
Member

if it's not enforced it'll get back or stay red forever. though conversely it has to be non flaky green to be enforced. I know you are fighting a tide there joy, but you're almost there... keep pushing!

@xiaolanz
Copy link
Copy Markdown
Contributor Author

Yeah, it's a deliberate art to find and exclude all flaky tests while identifying and fixing the real regressions in a timely fashion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants