test(proxy::config): deflake TestInitialSync by knight42 · Pull Request #94564 · kubernetes/kubernetes

knight42 · 2020-09-05T19:36:26Z

Signed-off-by: knight42 anonymousknight96@gmail.com

What type of PR is this?

/kind flake

What this PR does / why we need it:

https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/94556/pull-kubernetes-bazel-test/1302309891905425412

=== RUN   TestInitialSync
I0905 18:24:19.320048      23 config.go:315] Starting service config controller
I0905 18:24:19.320662      23 shared_informer.go:240] Waiting for caches to sync for service config
I0905 18:24:19.324410      23 shared_informer.go:247] Caches are synced for service config 
I0905 18:24:19.320602      23 config.go:133] Starting endpoints config controller
I0905 18:24:19.325287      23 shared_informer.go:240] Waiting for caches to sync for endpoints config
I0905 18:24:19.325457      23 shared_informer.go:247] Caches are synced for endpoints config 
    api_test.go:171: Unexpected endpoints: []*v1.Endpoints{(*v1.Endpoints)(0xc0000dc500)}, expected: []*v1.Endpoints{(*v1.Endpoints)(0xc0003e5cc0), (*v1.Endpoints)(0xc0003e5b80)}
--- FAIL: TestInitialSync (0.01s)

The error could be reproduced by the following patch:

diff --git pkg/proxy/config/config_test.go pkg/proxy/config/config_test.go
index 531b3b1a089..bec2aca56d9 100644
--- pkg/proxy/config/config_test.go
+++ pkg/proxy/config/config_test.go
@@ -23,7 +23,7 @@ import (
 	"testing"
 	"time"
 
-	"k8s.io/api/core/v1"
+	v1 "k8s.io/api/core/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/types"
 	"k8s.io/apimachinery/pkg/util/wait"
@@ -161,6 +161,7 @@ func NewEndpointsHandlerMock() *EndpointsHandlerMock {
 }
 
 func (h *EndpointsHandlerMock) OnEndpointsAdd(endpoints *v1.Endpoints) {
+	time.Sleep(time.Second)
 	h.lock.Lock()
 	defer h.lock.Unlock()
 	namespacedName := types.NamespacedName{Namespace: endpoints.Namespace, Name: endpoints.Name}

I think the root cause is even after the cache is synced, the execution of event handlers, such as OnEndpointsAdd, might not have completed yet, so the function in process field received unexpected endpoints.

Which issue(s) this PR fixes:

Part of #94528

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

knight42 · 2020-09-05T19:38:38Z

/cc @liggitt

liggitt · 2020-09-06T02:30:09Z

/cc @dcbw

knight42 · 2020-09-09T09:06:58Z

/test pull-kubernetes-e2e-gce-ubuntu-containerd

cmluciano

General question error verbosity

cmluciano · 2020-12-03T21:02:26Z

pkg/proxy/config/api_test.go

Do you have any objections to returning the more verbose error with details on the current versus expected not being equal? Is the more concise error suggesting that a timeout would be the only reason this would fail?

Do you have any objections to returning the more verbose error with details on the current versus expected not being equal

I think we could add some logging if what we got was different from the expected, but it might not be desired to return an error immediately as the poll would be interrupted.

Is the more concise error suggesting that a timeout would be the only reason this would fail?

yes

knight42 · 2020-12-08T03:53:31Z

I spotted this flake again https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/97083/pull-kubernetes-bazel-test/1336141738829418496

@dcbw would you mind taking a look?

fejta-bot · 2021-03-08T03:58:06Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

fejta-bot · 2021-04-07T04:40:37Z

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

fejta-bot · 2021-05-07T04:56:37Z

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

k8s-ci-robot · 2021-05-07T04:56:46Z

@fejta-bot: Closed this PR.

Details

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

knight42 · 2021-06-11T13:42:50Z

/remove-lifecycle rotten
/reopen

k8s-ci-robot · 2021-06-11T13:42:56Z

@knight42: Reopened this PR.

Details

In response to this:

/remove-lifecycle rotten
/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot · 2021-06-11T13:43:02Z

@knight42: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Signed-off-by: Jian Zeng <zengjian.zj@bytedance.com>

knight42 · 2021-06-11T15:05:28Z

/cc @aojea

aojea · 2021-06-11T23:00:20Z

/retest

[aojea@aojea-laptop kubernetes]$ go test -v -race ./pkg/proxy/config/ -run TestInitialSync
=== RUN   TestInitialSync
I0612 00:57:32.582516  145359 config.go:315] Starting service config controller
I0612 00:57:32.582718  145359 shared_informer.go:240] Waiting for caches to sync for service config
I0612 00:57:32.582809  145359 shared_informer.go:247] Caches are synced for service config 
I0612 00:57:32.582843  145359 config.go:133] Starting endpoints config controller
I0612 00:57:32.582861  145359 shared_informer.go:240] Waiting for caches to sync for endpoints config
I0612 00:57:32.582938  145359 shared_informer.go:247] Caches are synced for endpoints config 
--- PASS: TestInitialSync (0.01s)
PASS
ok      k8s.io/kubernetes/pkg/proxy/config      0.066s
[aojea@aojea-laptop kubernetes]$ stress ./config.test -test.run TestInitialSync -test.v
5s: 644 runs so far, 0 failures
10s: 1302 runs so far, 0 failures
15s: 1955 runs so far, 0 failures
20s: 2597 runs so far, 0 failures
25s: 3228 runs so far, 0 failures
30s: 3882 runs so far, 0 failures
35s: 4538 runs so far, 0 failures
40s: 5182 runs so far, 0 failures
45s: 5823 runs so far, 0 failures
50s: 6464 runs so far, 0 failures
55s: 7095 runs so far, 0 failures
1m0s: 7738 runs so far, 0 failures
1m5s: 8375 runs so far, 0 failures
1m10s: 9015 runs so far, 0 failures

aojea · 2021-06-11T23:14:10Z

/lgtm
/approve

k8s-ci-robot · 2021-06-11T23:14:33Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aojea, knight42

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~pkg/proxy/OWNERS~~ [aojea]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

BenTheElder · 2021-06-12T06:22:41Z

thank you!

k8s-ci-robot requested review from bowei and lavalamp September 5, 2020 19:37

k8s-ci-robot added sig/network Categorizes an issue or PR as relevant to SIG Network. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Sep 5, 2020

knight42 force-pushed the fix/TestInitialSync branch from d0394f0 to b148677 Compare September 5, 2020 19:38

k8s-ci-robot requested a review from liggitt September 5, 2020 19:38

liggitt removed request for lavalamp and liggitt September 6, 2020 02:30

k8s-ci-robot requested a review from dcbw September 6, 2020 02:30

knight42 force-pushed the fix/TestInitialSync branch from b148677 to 3e43e62 Compare September 6, 2020 05:54

cmluciano reviewed Dec 3, 2020

View reviewed changes

k8s-ci-robot added lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Mar 8, 2021

liggitt mentioned this pull request Mar 9, 2021

Promote PodDisruptionBudget to policy/v1 #99290

Merged

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Apr 7, 2021

k8s-ci-robot closed this May 7, 2021

knight42 mentioned this pull request Jun 11, 2021

pkg/proxy/config TestInitialSync flake #102792

Closed

k8s-ci-robot reopened this Jun 11, 2021

k8s-ci-robot added needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. and removed lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. labels Jun 11, 2021

knight42 force-pushed the fix/TestInitialSync branch from 3e43e62 to aa5563b Compare June 11, 2021 13:59

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 11, 2021

test(proxy::config): deflake TestInitialSync

9109d92

Signed-off-by: Jian Zeng <zengjian.zj@bytedance.com>

knight42 force-pushed the fix/TestInitialSync branch from aa5563b to 9109d92 Compare June 11, 2021 15:01

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jun 11, 2021

k8s-ci-robot requested a review from aojea June 11, 2021 15:05

k8s-ci-robot assigned aojea Jun 11, 2021

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 11, 2021

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 11, 2021

k8s-ci-robot merged commit 496f16c into kubernetes:master Jun 12, 2021

k8s-ci-robot added this to the v1.22 milestone Jun 12, 2021

knight42 deleted the fix/TestInitialSync branch June 12, 2021 02:09

Conversation

knight42 commented Sep 5, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

knight42 commented Sep 5, 2020

Uh oh!

liggitt commented Sep 6, 2020

Uh oh!

knight42 commented Sep 9, 2020

Uh oh!

cmluciano left a comment

Choose a reason for hiding this comment

Uh oh!

cmluciano Dec 3, 2020

Choose a reason for hiding this comment

Uh oh!

knight42 Dec 5, 2020

Choose a reason for hiding this comment

Uh oh!

knight42 commented Dec 8, 2020

Uh oh!

fejta-bot commented Mar 8, 2021

Uh oh!

fejta-bot commented Apr 7, 2021

Uh oh!

fejta-bot commented May 7, 2021

Uh oh!

k8s-ci-robot commented May 7, 2021

Uh oh!

knight42 commented Jun 11, 2021

Uh oh!

k8s-ci-robot commented Jun 11, 2021

Uh oh!

k8s-ci-robot commented Jun 11, 2021

Uh oh!

knight42 commented Jun 11, 2021

Uh oh!

aojea commented Jun 11, 2021

Uh oh!

aojea commented Jun 11, 2021

Uh oh!

k8s-ci-robot commented Jun 11, 2021

Uh oh!

BenTheElder commented Jun 12, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

knight42 commented Sep 5, 2020 •

edited

Loading