[18.09 backport] Fix update out of sequence and increase max recv gRPC message size for nodes and secrets by thaJeztah · Pull Request #2900 · moby/swarmkit

thaJeztah · 2019-09-25T09:39:06Z

built on top of #2836 ([18.09 backport] Switch to go 1.11) rebased

relates to

integration-cli: fix swarm tests flakiness moby#39531 integration-cli: fix swarm tests flakiness
[18.09 backport] integration-cli: fix swarm tests flakiness docker-archive/engine#346 [18.09 backport] integration-cli: fix swarm tests flakiness

Backports of

Increased wait time on test utils WaitForCluster and WatchTaskCreate #2762 Increased wait time on test utils WaitForCluster and WatchTaskCreate
Allow using Configs as CredentialSpecs #2771 Allow using Configs as CredentialSpecs
- second commit only (attempt to fix weirdly broken tests)
Fix flaky tests #2808 Fix flaky tests
Swap gometalinter for golangci-lint #2866 Swap gometalinter for golangci-lint
Increase max recv gRPC message size to initialize connection broker #2869 Increase max recv gRPC message size to initialize connection broker
related / similar to cluster: set bigger grpc limit for array requests moby#38103 / [18.06] cluster: set bigger grpc limit for array requests docker-archive/engine#102 cluster: set bigger grpc limit for array requests
related / similar to Increase max recv gRPC message size for nodes and secrets moby#39306 Increase max recv gRPC message size for nodes and secrets
fixes Error generated when messages size is too big #2733 Error generated when messages size is too big
Fix update out of sequence #2870 Fix update out of sequence

Cherry-pick were clean, no conflicts

# https://github.com/docker/swarmkit/pull/2762 Increased wait time on test utils WaitForCluster and WatchTaskCreate
git cherry-pick -s -S -x 5f167cab731ee75bc6cf3888fd2a4a5b5194f924

# second commit from https://github.com/docker/swarmkit/pull/2771 Allow using Configs as CredentialSpecs
git cherry-pick -s -S -x be26111c4a48c44fac04c17c69fd2504aea6db91

# https://github.com/docker/swarmkit/pull/2808 Fix flaky tests
git cherry-pick -s -S -x 06a356671bc11e4fd5d754f257f9c5f93ec5c563

# https://github.com/docker/swarmkit/pull/2866 Swap gometalinter for golangci-lint
git cherry-pick -s -S -x 27c2d27e23e76243bb49ec5d803a6e40b3f96f7a

# https://github.com/docker/swarmkit/pull/2869 Increase max recv gRPC message size to initialize connection broker
git cherry-pick -s -S -x 127e816ed8c8de4c981e02c9d0b45c70d3baa824

# https://github.com/docker/swarmkit/pull/2870 Fix update out of sequence
git cherry-pick -s -S -x d68ac46e3b11d7384472677d210bb0ce941284dc

thaJeztah · 2019-09-25T09:43:06Z

Argh; I suspect this is running Go 1.13, and related to golang/go#34073 / golang/go#31859

#!/bin/bash -eo pipefail
make check binaries checkprotos
🐳 check
ERRO Running error: context loading failed: failed to load program with go/packages: unsupported version of go: exit status 2: flag provided but not defined: -compiled
usage: list [-e] [-f format] [-json] [build flags] [packages]
Run 'go help list' for details.
 
make: *** [check] Error 3
Exited with code 2

thaJeztah · 2019-09-25T09:44:21Z

Oh... the reverse; it's still on Go 1.10 😞

thaJeztah · 2019-09-25T09:55:47Z

rebased on top of #2836; marked as WIP and will rebase once that's merged

codecov · 2019-09-25T10:04:43Z

Codecov Report

Merging #2900 into bump_v18.09 will decrease coverage by 0.03%.
The diff coverage is 31.25%.

@@               Coverage Diff               @@
##           bump_v18.09    #2900      +/-   ##
===============================================
- Coverage        61.79%   61.75%   -0.04%     
===============================================
  Files              134      134              
  Lines            21869    21875       +6     
===============================================
- Hits             13513    13508       -5     
- Misses            6894     6912      +18     
+ Partials          1462     1455       -7

thaJeztah · 2019-09-25T10:19:57Z

Failure is a flaky test; #2559

--- FAIL: TestUpdaterRollback (35.31s)
    --- FAIL: TestUpdaterRollback/pause/monitor_set/spec_version_unset (3.61s)
        update_test.go:272: service was updated
	Error Trace:	testutils.go:42
			update_test.go:343
			update_test.go:20
	Error:		no task creation
            

    --- FAIL: TestUpdaterRollback/continue/spec_version_set (30.00s)
        update_test.go:272: service was updated
        update_test.go:324: test timed out before watchServiceUpdate provided an event
FAIL
coverage: 84.9% of statements
FAIL	github.com/docker/swarmkit/manager/orchestrator/replicated	37.618s
make: *** [coverage] Error 1
Exited with code 2

I'll try restarting CI

Signed-off-by: Olli Janatuinen <olli.janatuinen@gmail.com> (cherry picked from commit 5f167ca) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

Signed-off-by: Drew Erny <drew.erny@docker.com> (cherry picked from commit be26111) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

It is likely that a large portion of test flakiness, especially in CI, comes from the fact that swarmkit components under test are started in goroutines, but those goroutines never have an opportunity to run. This adds code ensuring those goroutines are scheduled and run, which should hopefully solve many inexplicably flaky tests. Additionally, increased test timeouts, to hopefully cover a few more flaky cases. Finally, removed direct use of the atomic package, in favor of less efficient but higher-level mutexes. Signed-off-by: Drew Erny <drew.erny@docker.com> (cherry picked from commit 06a3566) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

gometalinter is deprecated, and golangci-lint is its recommended successor. This commit adds golangci-lint as the linter for swarmkit. In addition, golangci-lint found a few issues in the code that were not yet identified, and so those issues have been fixed. Signed-off-by: Drew Erny <drew.erny@docker.com> (cherry picked from commit 27c2d27) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

Signed-off-by: nmengin <nicolas@containo.us> (cherry picked from commit 127e816) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

A simple but old error has recently become evident. Due to the fact that we read an object and then write it back across the boundaries of a transaction, it is possible for the task object to have changed in between transactions. This would cause the attempt to write out the old task to suffer an "Update out of sequence" error. This fix simply reads the latest version of the task back out within the boundary of a transaction to avoid the race. Signed-off-by: Drew Erny <drew.erny@docker.com> (cherry picked from commit d68ac46) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

thaJeztah · 2019-10-01T16:36:21Z

rebased 👍

thaJeztah · 2019-10-01T17:03:23Z

@dperny this one's green, and probably ready to go

…v18.09) full diff: moby/swarmkit@142a737...5c86095 - moby/swarmkit#2892 [18.09 backport] Remove hardcoded IPAM config subnet value for ingress network - backport of moby/swarmkit#2890 Remove hardcoded IPAM config subnet value for ingress network - fixes [ENGORC-2651](https://docker.atlassian.net/browse/ENGORC-2651) - moby/swarmkit#2836 [18.09 backport] Switch to go 1.11 - backport of moby/swarmkit#2752 Switch to go 1.11 - moby/swarmkit#2901 [18.09 backport] Bump to golang 1.12.9 - backport of moby/swarmkit#2880 Bump to golang 1.12.9 - moby/swarmkit#2900 [18.09 backport] Fix update out of sequence and increase max recv gRPC message size for nodes and secrets - backport of moby/swarmkit#2762 Increased wait time on test utils WaitForCluster and WatchTaskCreate - backport of moby/swarmkit#2771 Allow using Configs as CredentialSpecs - **second commit only** (attempt to fix weirdly broken tests) - backport of moby/swarmkit#2808 Fix flaky tests - backport of moby/swarmkit#2866 Swap gometalinter for golangci-lint - backport of moby/swarmkit#2869 Increase max recv gRPC message size to initialize connection broker - related / similar to moby#38103 / docker-archive#102 cluster: set bigger grpc limit for array requests - related / similar to moby#39306 Increase max recv gRPC message size for nodes and secrets - fixes moby/swarmkit#2733 Error generated when messages size is too big - backport of moby/swarmkit#2870 Fix update out of sequence Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

…v18.09) full diff: moby/swarmkit@142a737...5c86095 - moby/swarmkit#2892 [18.09 backport] Remove hardcoded IPAM config subnet value for ingress network - backport of moby/swarmkit#2890 Remove hardcoded IPAM config subnet value for ingress network - fixes [ENGORC-2651](https://docker.atlassian.net/browse/ENGORC-2651) - moby/swarmkit#2836 [18.09 backport] Switch to go 1.11 - backport of moby/swarmkit#2752 Switch to go 1.11 - moby/swarmkit#2901 [18.09 backport] Bump to golang 1.12.9 - backport of moby/swarmkit#2880 Bump to golang 1.12.9 - moby/swarmkit#2900 [18.09 backport] Fix update out of sequence and increase max recv gRPC message size for nodes and secrets - backport of moby/swarmkit#2762 Increased wait time on test utils WaitForCluster and WatchTaskCreate - backport of moby/swarmkit#2771 Allow using Configs as CredentialSpecs - **second commit only** (attempt to fix weirdly broken tests) - backport of moby/swarmkit#2808 Fix flaky tests - backport of moby/swarmkit#2866 Swap gometalinter for golangci-lint - backport of moby/swarmkit#2869 Increase max recv gRPC message size to initialize connection broker - related / similar to moby/moby#38103 / docker-archive/engine#102 cluster: set bigger grpc limit for array requests - related / similar to moby/moby#39306 Increase max recv gRPC message size for nodes and secrets - fixes moby/swarmkit#2733 Error generated when messages size is too big - backport of moby/swarmkit#2870 Fix update out of sequence Signed-off-by: Sebastiaan van Stijn <github@gone.nl> Upstream-commit: e06f07ef337ab890f211397d6b408b75a2512dc5 Component: engine

thaJeztah force-pushed the 18.09_backport_fix_update_out_of_sequence branch from a60a21a to cde561a Compare September 25, 2019 09:54

thaJeztah changed the title ~~[18.09 backport] Fix update out of sequence and increase max recv gRPC message size for nodes and secrets~~ [WIP][18.09 backport] Fix update out of sequence and increase max recv gRPC message size for nodes and secrets Sep 25, 2019

thaJeztah mentioned this pull request Sep 25, 2019

[18.09 backport] integration-cli: fix swarm tests flakiness docker-archive/engine#346

Merged

thaJeztah force-pushed the 18.09_backport_fix_update_out_of_sequence branch from cde561a to 1539b14 Compare September 25, 2019 10:03

thaJeztah force-pushed the 18.09_backport_fix_update_out_of_sequence branch from 1539b14 to 7c20388 Compare September 26, 2019 16:12

olljanat and others added 6 commits October 1, 2019 18:35

Increased wait time on test utils WaitForCluster and WatchTaskCreate

6a454da

Signed-off-by: Olli Janatuinen <olli.janatuinen@gmail.com> (cherry picked from commit 5f167ca) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

attempt to fix weirdly broken tests

6a2573e

Signed-off-by: Drew Erny <drew.erny@docker.com> (cherry picked from commit be26111) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

Set bigger grpc value to initialize connection broker

177caa5

Signed-off-by: nmengin <nicolas@containo.us> (cherry picked from commit 127e816) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

thaJeztah force-pushed the 18.09_backport_fix_update_out_of_sequence branch from 7c20388 to bfce89e Compare October 1, 2019 16:35

thaJeztah changed the title ~~[WIP][18.09 backport] Fix update out of sequence and increase max recv gRPC message size for nodes and secrets~~ [18.09 backport] Fix update out of sequence and increase max recv gRPC message size for nodes and secrets Oct 1, 2019

dperny merged commit 5c86095 into moby:bump_v18.09 Oct 2, 2019

thaJeztah deleted the 18.09_backport_fix_update_out_of_sequence branch October 2, 2019 21:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[18.09 backport] Fix update out of sequence and increase max recv gRPC message size for nodes and secrets#2900

[18.09 backport] Fix update out of sequence and increase max recv gRPC message size for nodes and secrets#2900
dperny merged 6 commits intomoby:bump_v18.09from
thaJeztah:18.09_backport_fix_update_out_of_sequence

thaJeztah commented Sep 25, 2019 •

edited

Loading

Uh oh!

thaJeztah commented Sep 25, 2019

Uh oh!

thaJeztah commented Sep 25, 2019

Uh oh!

thaJeztah commented Sep 25, 2019 •

edited

Loading

Uh oh!

codecov bot commented Sep 25, 2019 •

edited

Loading

Uh oh!

thaJeztah commented Sep 25, 2019

Uh oh!

thaJeztah commented Oct 1, 2019

Uh oh!

thaJeztah commented Oct 1, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

thaJeztah commented Sep 25, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

built on top of #2836 ([18.09 backport] Switch to go 1.11) rebased

Uh oh!

thaJeztah commented Sep 25, 2019

Uh oh!

thaJeztah commented Sep 25, 2019

Uh oh!

thaJeztah commented Sep 25, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Sep 25, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

thaJeztah commented Sep 25, 2019

Uh oh!

thaJeztah commented Oct 1, 2019

Uh oh!

thaJeztah commented Oct 1, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

thaJeztah commented Sep 25, 2019 •

edited

Loading

thaJeztah commented Sep 25, 2019 •

edited

Loading

codecov bot commented Sep 25, 2019 •

edited

Loading